[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:18:25 dist-test-slave-btw8 krb5kdc[20452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:18:25 dist-test-slave-btw8 krb5kdc[20452](info): set up 2 sockets
Aug 10 03:18:25 dist-test-slave-btw8 krb5kdc[20452](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 03:18:26.749039 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:18:26 dist-test-slave-btw8 krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:18:26.981264 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41293
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:27.356609 20467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:27.357030 20467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:27.357196 20467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:27.365136 20467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:18:27.365280 20467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:27.365389 20467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:27.365522 20467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:18:27.365612 20467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:18:27.377031 20467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:27.379293 20467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:27.381381 20467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:27.411322 20472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:27.411909 20475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:27.412011 20473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:27.413991 20467 server_base.cc:1034] running on GCE node
I20260810 03:18:27.415416 20467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:27.418032 20467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:27.419533 20467 hybrid_clock.cc:648] HybridClock initialized: now 1786331907419406 us; error 130 us; skew 500 ppm
I20260810 03:18:27.420215 20467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:27 dist-test-slave-btw8 krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331907, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:27.424309 20467 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:18:27.427035 20467 webserver.cc:460] Webserver started at http://127.19.246.254:40829/ using document root <none> and password file <none>
I20260810 03:18:27.428265 20467 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:27.428440 20467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:27.428951 20467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:27.433538 20467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "4621784728c841d0b20549b1939fcd6b"
format_stamp: "Formatted at 2026-08-10 03:18:27 on dist-test-slave-btw8"
server_key: "4ca6f896c53daeb8bd4f346253deb86f"
server_key_iv: "d28421df08e1098223fafbbca035f826"
server_key_version: "encryptionkey@0"
I20260810 03:18:27.434823 20467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "4621784728c841d0b20549b1939fcd6b"
format_stamp: "Formatted at 2026-08-10 03:18:27 on dist-test-slave-btw8"
server_key: "4ca6f896c53daeb8bd4f346253deb86f"
server_key_iv: "d28421df08e1098223fafbbca035f826"
server_key_version: "encryptionkey@0"
I20260810 03:18:27.444393 20467 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260810 03:18:27.450920 20482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:27.453441 20467 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260810 03:18:27.453869 20467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "4621784728c841d0b20549b1939fcd6b"
format_stamp: "Formatted at 2026-08-10 03:18:27 on dist-test-slave-btw8"
server_key: "4ca6f896c53daeb8bd4f346253deb86f"
server_key_iv: "d28421df08e1098223fafbbca035f826"
server_key_version: "encryptionkey@0"
I20260810 03:18:27.454146 20467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:27.473702 20467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:27.480027 20467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:27.480507 20467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:27.505090 20467 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:41293
I20260810 03:18:27.505129 20540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:41293 every 8 connection(s)
I20260810 03:18:27.507846 20467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:18:27.515590 20541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:27.517732 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 20467
I20260810 03:18:27.518172 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:18:27.520663 20443 external_mini_cluster.cc:1428] Setting key 668cd2bcef17849297651e4879f49245
I20260810 03:18:27.534797 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b: Bootstrap starting.
I20260810 03:18:27.540998 20541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:27.543083 20541 log.cc:824] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:27.547181 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b: No bootstrap required, opened a new log
Aug 10 03:18:27 dist-test-slave-btw8 krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:27.557446 20541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4621784728c841d0b20549b1939fcd6b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41293 } }
I20260810 03:18:27.558226 20541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:27.558435 20541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4621784728c841d0b20549b1939fcd6b, State: Initialized, Role: FOLLOWER
I20260810 03:18:27.560025 20541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4621784728c841d0b20549b1939fcd6b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41293 } }
I20260810 03:18:27.560487 20541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:18:27.560839 20541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:18:27.561102 20541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:27.564229 20541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4621784728c841d0b20549b1939fcd6b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41293 } }
I20260810 03:18:27.565575 20541 leader_election.cc:302] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4621784728c841d0b20549b1939fcd6b; no voters:
I20260810 03:18:27.567781 20541 leader_election.cc:288] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:18:27.568086 20546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:27.574020 20541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:18:27.574334 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:27.527696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.19.246.254:41293)
0810 03:18:27.529085 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:27.529112 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:27.530615 (+ 1503us) server_negotiation.cc:407] Connection header received
0810 03:18:27.541429 (+ 10814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:27.541452 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:27.542133 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:27.542630 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:27.546138 (+ 3508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:27.547934 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:27.549853 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:27.550699 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:27.551049 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:27.558762 (+ 7713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:27.558893 (+ 131us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:27.558949 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:27.559013 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:27.563082 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:27.564640 (+ 1558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:27.564650 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:27.564658 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:27.565128 (+ 470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:27.566019 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:27.566025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:27.566029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:27.567306 (+ 1277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:27.567891 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:27.571879 (+ 3988us) server_negotiation.cc:299] Negotiation successful
0810 03:18:27.572517 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":518,"threads_started":1}
I20260810 03:18:27.576403 20546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [term 1 LEADER]: Becoming Leader. State: Replica: 4621784728c841d0b20549b1939fcd6b, State: Running, Role: LEADER
I20260810 03:18:27.577651 20546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4621784728c841d0b20549b1939fcd6b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41293 } }
I20260810 03:18:27.585356 20548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4621784728c841d0b20549b1939fcd6b. Latest consensus state: current_term: 1 leader_uuid: "4621784728c841d0b20549b1939fcd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4621784728c841d0b20549b1939fcd6b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41293 } } }
I20260810 03:18:27.586035 20548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:27.587034 20547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4621784728c841d0b20549b1939fcd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4621784728c841d0b20549b1939fcd6b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41293 } } }
I20260810 03:18:27.587383 20547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:27.593137 20558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:18:27.601480 20558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:18:27.616660 20558 catalog_manager.cc:1347] Generated new cluster ID: 5c0c5b285eaa42d099ee56048f9539b0
I20260810 03:18:27.616801 20558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:18:27.630506 20558 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:18:27.632579 20558 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:18:27.645776 20558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4621784728c841d0b20549b1939fcd6b: Generated new TSK 0
I20260810 03:18:27.647339 20558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:18:27.724804 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41293
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:28.098884 20567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:28.099196 20567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:28.099416 20567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:28.106948 20567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:28.107085 20567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:28.107275 20567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:18:28.117678 20567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:28.119073 20567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:28.120494 20567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:28.135372 20574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:28.135289 20576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:28.135622 20572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:28.136603 20567 server_base.cc:1034] running on GCE node
I20260810 03:18:28.137274 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:28.138352 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:28.139556 20567 hybrid_clock.cc:648] HybridClock initialized: now 1786331908139507 us; error 33 us; skew 500 ppm
I20260810 03:18:28.139984 20567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:28 dist-test-slave-btw8 krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331908, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:28.143612 20567 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:18:28.145498 20567 webserver.cc:460] Webserver started at http://127.19.246.193:41697/ using document root <none> and password file <none>
I20260810 03:18:28.146268 20567 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:28.146409 20567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:28.146772 20567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:28.150445 20567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "6443c21d59604b1186cfd7a2fba7e4c7"
format_stamp: "Formatted at 2026-08-10 03:18:28 on dist-test-slave-btw8"
server_key: "ed7c09e4d99ccacce263cd4a14f5330a"
server_key_iv: "b1a85a857a71167928878eac045dd183"
server_key_version: "encryptionkey@0"
I20260810 03:18:28.151311 20567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "6443c21d59604b1186cfd7a2fba7e4c7"
format_stamp: "Formatted at 2026-08-10 03:18:28 on dist-test-slave-btw8"
server_key: "ed7c09e4d99ccacce263cd4a14f5330a"
server_key_iv: "b1a85a857a71167928878eac045dd183"
server_key_version: "encryptionkey@0"
I20260810 03:18:28.157117 20567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 03:18:28.161499 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:28.163116 20567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:18:28.163336 20567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "6443c21d59604b1186cfd7a2fba7e4c7"
format_stamp: "Formatted at 2026-08-10 03:18:28 on dist-test-slave-btw8"
server_key: "ed7c09e4d99ccacce263cd4a14f5330a"
server_key_iv: "b1a85a857a71167928878eac045dd183"
server_key_version: "encryptionkey@0"
I20260810 03:18:28.163693 20567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:28.195230 20567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:28.200186 20567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:28.200469 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:28.201581 20567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:28.203310 20567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:28.203452 20567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 03:18:28.203636 20567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:28.203691 20567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:28.244848 20567 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:43173
I20260810 03:18:28.244912 20703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:43173 every 8 connection(s)
I20260810 03:18:28.246299 20567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:18:28.251332 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 20567
I20260810 03:18:28.251711 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:18:28.252425 20443 external_mini_cluster.cc:1428] Setting key c75623cef3b6e0e6c849e7603edf1920
Aug 10 03:18:28 dist-test-slave-btw8 krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331908, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:28.275446 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:28.250139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:36025 (local address 127.19.246.254:41293)
0810 03:18:28.250637 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:28.250642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:28.252598 (+ 1956us) server_negotiation.cc:407] Connection header received
0810 03:18:28.254234 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:28.254240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:28.254347 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:28.254500 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:28.256887 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:28.257728 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:28.261671 (+ 3943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:28.262144 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:28.262287 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:28.267985 (+ 5698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:28.267996 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:28.268004 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:28.268063 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:28.271774 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:28.272705 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:28.272718 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:28.272721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:28.272860 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:28.273333 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:28.273340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:28.273349 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:28.273631 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:28.273815 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:28.274736 (+ 921us) server_negotiation.cc:299] Negotiation successful
0810 03:18:28.274981 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":241,"threads_started":1}
I20260810 03:18:28.276523 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:28.250807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41293 (local address 127.19.246.193:36025)
0810 03:18:28.251733 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:28.251781 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:28.253916 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:28.254774 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:28.254786 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:28.255667 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:28.256616 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:28.256649 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:28.257988 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:28.257995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:28.261412 (+ 3417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:28.261434 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:28.262430 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:28.262440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:28.262900 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:28.264213 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:28.264251 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:28.267623 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:28.271995 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:28.272012 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:28.272028 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:28.272545 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:28.272988 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:28.272993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:28.273003 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:28.273204 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:28.273771 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:28.273781 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:28.274245 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 03:18:28.274623 (+ 378us) client_negotiation.cc:240] Negotiation successful
0810 03:18:28.275220 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":402,"threads_started":1}
I20260810 03:18:28.278584 20704 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41293
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
I20260810 03:18:28.279165 20704 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:28.280403 20704 heartbeater.cc:511] Master 127.19.246.254:41293 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
I20260810 03:18:28.284999 20501 ts_manager.cc:194] Registered new tserver with Master: 6443c21d59604b1186cfd7a2fba7e4c7 (127.19.246.193:43173)
I20260810 03:18:28.287429 20501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:36025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:18:28.311501 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41293
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:28.672778 20711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:28.673089 20711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:28.673255 20711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:28.681147 20711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:28.681278 20711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:28.681495 20711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:18:28.691793 20711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:28.693198 20711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:28.694566 20711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:28.708925 20722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:28.709175 20718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:28.709100 20719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:28.710328 20711 server_base.cc:1034] running on GCE node
I20260810 03:18:28.711130 20711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:28.712425 20711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:28.713747 20711 hybrid_clock.cc:648] HybridClock initialized: now 1786331908713668 us; error 64 us; skew 500 ppm
I20260810 03:18:28.714171 20711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:28 dist-test-slave-btw8 krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331908, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:28.718173 20711 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:18:28.720235 20711 webserver.cc:460] Webserver started at http://127.19.246.194:42285/ using document root <none> and password file <none>
I20260810 03:18:28.721029 20711 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:28.721210 20711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:28.721585 20711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:28.725055 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "eac3109430f9476894ea13481e23c06a"
format_stamp: "Formatted at 2026-08-10 03:18:28 on dist-test-slave-btw8"
server_key: "1ef88262daf1426d435f8b9eb53872bb"
server_key_iv: "b2a83f2200d1fc0d2f4d5c2943cafbba"
server_key_version: "encryptionkey@0"
I20260810 03:18:28.725936 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "eac3109430f9476894ea13481e23c06a"
format_stamp: "Formatted at 2026-08-10 03:18:28 on dist-test-slave-btw8"
server_key: "1ef88262daf1426d435f8b9eb53872bb"
server_key_iv: "b2a83f2200d1fc0d2f4d5c2943cafbba"
server_key_version: "encryptionkey@0"
I20260810 03:18:28.732007 20711 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 03:18:28.735954 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:28.737505 20711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:18:28.737775 20711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "eac3109430f9476894ea13481e23c06a"
format_stamp: "Formatted at 2026-08-10 03:18:28 on dist-test-slave-btw8"
server_key: "1ef88262daf1426d435f8b9eb53872bb"
server_key_iv: "b2a83f2200d1fc0d2f4d5c2943cafbba"
server_key_version: "encryptionkey@0"
I20260810 03:18:28.738010 20711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:28.769475 20711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:28.774406 20711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:28.774765 20711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:28.775990 20711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:28.777549 20711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:28.777654 20711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:28.777828 20711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:28.777915 20711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:28.821892 20711 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:39341
I20260810 03:18:28.821955 20848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:39341 every 8 connection(s)
I20260810 03:18:28.823413 20711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:18:28.825467 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 20711
I20260810 03:18:28.825687 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:18:28.826190 20443 external_mini_cluster.cc:1428] Setting key 34d2a848f0db68476975a1b49f125891
Aug 10 03:18:28 dist-test-slave-btw8 krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331908, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:28.850238 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:28.826983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:42845 (local address 127.19.246.254:41293)
0810 03:18:28.827625 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:28.827631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:28.828751 (+ 1120us) server_negotiation.cc:407] Connection header received
0810 03:18:28.830876 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:28.830891 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:28.831071 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:28.831315 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:28.833060 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:28.834885 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:28.836725 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:28.837546 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:28.837738 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:28.842377 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:28.842382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:28.842386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:28.842425 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:28.846300 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:28.847224 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:28.847234 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:28.847241 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:28.847452 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:28.847989 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:28.847995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:28.848000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:28.848440 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:28.848630 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:28.849507 (+ 877us) server_negotiation.cc:299] Negotiation successful
0810 03:18:28.849787 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":336,"threads_started":1}
I20260810 03:18:28.851128 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:28.827704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41293 (local address 127.19.246.194:42845)
0810 03:18:28.828586 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:28.828639 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:28.830527 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:28.831595 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:28.831605 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:28.832193 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:28.832802 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:28.832819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:28.835077 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:28.835081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:28.836551 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:28.836563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:28.837798 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:28.837803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:28.838036 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:28.838894 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:28.838931 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:28.842074 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:28.846503 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:28.846511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:28.846527 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:28.847054 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:28.847605 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:28.847610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:28.847621 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:28.847845 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:28.848629 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:28.848636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:28.849055 (+ 419us) client_negotiation.cc:769] Sending connection context
0810 03:18:28.849360 (+ 305us) client_negotiation.cc:240] Negotiation successful
0810 03:18:28.849823 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":413,"threads_started":1}
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
I20260810 03:18:28.852960 20849 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41293
I20260810 03:18:28.853284 20849 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:28.854094 20849 heartbeater.cc:511] Master 127.19.246.254:41293 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
I20260810 03:18:28.855953 20501 ts_manager.cc:194] Registered new tserver with Master: eac3109430f9476894ea13481e23c06a (127.19.246.194:39341)
I20260810 03:18:28.857544 20501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:42845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:18:28.884675 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41293
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:29.252461 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:29.252857 20855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:29.253044 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:29.262457 20855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:29.262706 20855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:29.262938 20855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:18:29.275089 20855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:29.276659 20855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:29.278316 20855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:18:29.292666 20704 heartbeater.cc:503] Master 127.19.246.254:41293 was elected leader, sending a full tablet report...
W20260810 03:18:29.293870 20861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:29.294050 20862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:29.294543 20864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:29.296108 20855 server_base.cc:1034] running on GCE node
I20260810 03:18:29.296875 20855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:29.297978 20855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:29.299232 20855 hybrid_clock.cc:648] HybridClock initialized: now 1786331909299172 us; error 54 us; skew 500 ppm
I20260810 03:18:29.299680 20855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:29 dist-test-slave-btw8 krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331909, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:29.303118 20855 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:18:29.304940 20855 webserver.cc:460] Webserver started at http://127.19.246.195:34293/ using document root <none> and password file <none>
I20260810 03:18:29.305766 20855 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:29.305924 20855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:29.306269 20855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:29.309492 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "01d96a5c2aba4e21ad5ca7a3e76fbffc"
format_stamp: "Formatted at 2026-08-10 03:18:29 on dist-test-slave-btw8"
server_key: "21e38c85deed41422d2ac17d7dd8c845"
server_key_iv: "b8c7ce67bcfa15e682ca6978cc6549df"
server_key_version: "encryptionkey@0"
I20260810 03:18:29.310356 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "01d96a5c2aba4e21ad5ca7a3e76fbffc"
format_stamp: "Formatted at 2026-08-10 03:18:29 on dist-test-slave-btw8"
server_key: "21e38c85deed41422d2ac17d7dd8c845"
server_key_iv: "b8c7ce67bcfa15e682ca6978cc6549df"
server_key_version: "encryptionkey@0"
I20260810 03:18:29.316284 20855 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 03:18:29.320196 20872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:29.321722 20855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:18:29.321933 20855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "01d96a5c2aba4e21ad5ca7a3e76fbffc"
format_stamp: "Formatted at 2026-08-10 03:18:29 on dist-test-slave-btw8"
server_key: "21e38c85deed41422d2ac17d7dd8c845"
server_key_iv: "b8c7ce67bcfa15e682ca6978cc6549df"
server_key_version: "encryptionkey@0"
I20260810 03:18:29.322165 20855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:29.339691 20855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:29.345013 20855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:29.345387 20855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:29.346570 20855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:29.348191 20855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:29.348306 20855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:29.348474 20855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:29.348572 20855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:29.389380 20855 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:43475
I20260810 03:18:29.389400 20990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:43475 every 8 connection(s)
I20260810 03:18:29.390916 20855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:18:29.397936 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 20855
I20260810 03:18:29.398180 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:18:29.398638 20443 external_mini_cluster.cc:1428] Setting key 0bc9a6aff4c76b680700eb5757f2e26f
Aug 10 03:18:29 dist-test-slave-btw8 krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331909, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:29.412995 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:29.393813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:33377 (local address 127.19.246.254:41293)
0810 03:18:29.394380 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:29.394387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:29.395307 (+ 920us) server_negotiation.cc:407] Connection header received
0810 03:18:29.397246 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:29.397251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:29.397322 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:29.397424 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:29.399287 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.400545 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.402318 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.402884 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.403029 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:29.406915 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:29.406921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:29.406924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:29.406958 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:29.409931 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.410721 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.410725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.410727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.410858 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.411268 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.411271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.411273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.411665 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:29.411792 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:29.412498 (+ 706us) server_negotiation.cc:299] Negotiation successful
0810 03:18:29.412702 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":279,"threads_started":1}
I20260810 03:18:29.414181 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:29.394326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41293 (local address 127.19.246.195:33377)
0810 03:18:29.395125 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:29.395165 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:29.396980 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:29.397678 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:29.397687 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:29.398429 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:29.399089 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:29.399106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.400746 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.400750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:29.402163 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:29.402176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.403129 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.403135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:29.403494 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:29.404278 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:29.404303 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:29.406684 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:29.410093 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:29.410102 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:29.410118 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:29.410609 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:29.410959 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:29.410962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:29.410964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:29.411164 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:29.411788 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:29.411795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:29.412157 (+ 362us) client_negotiation.cc:769] Sending connection context
0810 03:18:29.412402 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 03:18:29.412829 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":322,"threads_started":1}
I20260810 03:18:29.415879 20991 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41293
I20260810 03:18:29.416293 20991 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:29.417133 20991 heartbeater.cc:511] Master 127.19.246.254:41293 requested a full tablet report, sending...
I20260810 03:18:29.418839 20501 ts_manager.cc:194] Registered new tserver with Master: 01d96a5c2aba4e21ad5ca7a3e76fbffc (127.19.246.195:43475)
I20260810 03:18:29.419816 20501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:33377
I20260810 03:18:29.429821 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:18:29 dist-test-slave-btw8 krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:18:29 dist-test-slave-btw8 krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:18:29.485806 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:29.467819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.19.246.193:43173)
0810 03:18:29.468557 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:29.468564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:29.468595 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:18:29.468660 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:29.468667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:29.468922 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:29.469114 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:29.471022 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.472288 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.473663 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.474260 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.474367 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:29.478918 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:29.478934 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:29.478952 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:29.479000 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:29.482375 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.483263 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.483270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.483273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.483593 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.484049 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.484054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.484056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.484493 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:29.484735 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:29.485119 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 03:18:29.485337 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":400,"threads_started":1}
W20260810 03:18:29.487411 20617 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33836
Aug 10 03:18:29 dist-test-slave-btw8 krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:18:29 dist-test-slave-btw8 krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:18:29.541745 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:29.525675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.19.246.193:43173)
0810 03:18:29.525939 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:29.525949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:29.526191 (+ 242us) server_negotiation.cc:407] Connection header received
0810 03:18:29.526420 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:29.526426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:29.526567 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:29.526724 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:29.528283 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.529036 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.530404 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.530855 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.530959 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:29.535611 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:29.535617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:29.535621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:29.535665 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:29.538556 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.539447 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.539452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.539454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.539676 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.540124 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.540128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.540131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.540429 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:29.540656 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:29.540974 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:18:29.541314 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 03:18:29.544420 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 20567
I20260810 03:18:29.559509 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:29.545654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.19.246.193:43173)
0810 03:18:29.545931 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:29.545938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:29.546186 (+ 248us) server_negotiation.cc:407] Connection header received
0810 03:18:29.546386 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:29.546392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:29.546461 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:29.546681 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:29.548072 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.548821 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.550184 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.550632 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.550774 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:29.553342 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:29.553347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:29.553351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:29.553386 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:29.556436 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.557298 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.557304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.557309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.557466 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.557858 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.557864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.557867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.558154 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:29.558418 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:29.558768 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 03:18:29.559037 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 03:18:29.560679 20699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:29.653719 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 20567
I20260810 03:18:29.673537 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 20711
Aug 10 03:18:29 dist-test-slave-btw8 krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:18:29.691529 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:29.674643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.19.246.194:39341)
0810 03:18:29.675273 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:29.675281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:29.675310 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:18:29.675420 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:29.675429 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:29.675672 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:29.675858 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:29.677111 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.678184 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.679612 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.680266 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.680414 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:29.684942 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:29.684956 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:29.684972 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:29.685013 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:29.688228 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.689070 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.689080 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.689084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.689220 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.689704 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.689710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.689715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.690091 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:29.690420 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:29.690754 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 03:18:29.691044 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":279,"threads_started":1}
I20260810 03:18:29.692987 20844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:29.779608 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 20711
I20260810 03:18:29.799194 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 20855
Aug 10 03:18:29 dist-test-slave-btw8 krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:18:29.817385 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:29.800460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.19.246.195:43475)
0810 03:18:29.801231 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:29.801238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:29.801270 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:18:29.801331 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:29.801337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:29.801550 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:29.801735 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:29.803150 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.804291 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.805668 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:29.806253 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:29.806402 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:29.810977 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:29.810991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:29.811007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:29.811045 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:29.814208 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.815004 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.815013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.815015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.815162 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:29.815645 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:29.815651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:29.815656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:29.816031 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:29.816386 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:29.816707 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 03:18:29.817005 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":461,"threads_started":1}
I20260810 03:18:29.818804 20986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:29.905607 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 20855
I20260810 03:18:29.926055 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 20467
I20260810 03:18:29.927500 20535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:30.033952 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 20467
2026-08-10T03:18:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4530 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:18:30 dist-test-slave-btw8 krb5kdc[21014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:18:30 dist-test-slave-btw8 krb5kdc[21014](info): set up 2 sockets
Aug 10 03:18:30 dist-test-slave-btw8 krb5kdc[21014](info): commencing operation
krb5kdc: starting...
W20260810 03:18:31.207115 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:18:31 dist-test-slave-btw8 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:18:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:18:31.440398 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:38019
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:38019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:31.795483 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:31.795845 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:31.796026 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:31.803684 21027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:18:31.803817 21027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:31.803900 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:31.803992 21027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:18:31.804088 21027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:18:31.814745 21027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:38019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:38019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:31.816293 21027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:31.817739 21027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:31.830391 21032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:31.830497 21033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:31.830497 21035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:31.831452 21027 server_base.cc:1034] running on GCE node
I20260810 03:18:31.832406 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:31.833994 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:31.835224 21027 hybrid_clock.cc:648] HybridClock initialized: now 1786331911835172 us; error 51 us; skew 500 ppm
I20260810 03:18:31.835651 21027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:31 dist-test-slave-btw8 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331911, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:31.844210 21027 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:18:31.846182 21027 webserver.cc:460] Webserver started at http://127.19.246.254:34529/ using document root <none> and password file <none>
I20260810 03:18:31.847153 21027 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:31.847292 21027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:31.847728 21027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:31.851287 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "8dd3ec0daf1b45c499e531861db05b8b"
format_stamp: "Formatted at 2026-08-10 03:18:31 on dist-test-slave-btw8"
server_key: "eddaf2ef63b6c2d43da246cc59ab77a6"
server_key_iv: "844080b5cef1ed4f7ebb42714212a838"
server_key_version: "encryptionkey@0"
I20260810 03:18:31.852265 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "8dd3ec0daf1b45c499e531861db05b8b"
format_stamp: "Formatted at 2026-08-10 03:18:31 on dist-test-slave-btw8"
server_key: "eddaf2ef63b6c2d43da246cc59ab77a6"
server_key_iv: "844080b5cef1ed4f7ebb42714212a838"
server_key_version: "encryptionkey@0"
I20260810 03:18:31.858625 21027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:18:31.862670 21042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:31.864295 21027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:18:31.864471 21027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "8dd3ec0daf1b45c499e531861db05b8b"
format_stamp: "Formatted at 2026-08-10 03:18:31 on dist-test-slave-btw8"
server_key: "eddaf2ef63b6c2d43da246cc59ab77a6"
server_key_iv: "844080b5cef1ed4f7ebb42714212a838"
server_key_version: "encryptionkey@0"
I20260810 03:18:31.864774 21027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:31.884949 21027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:31.890782 21027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:31.891105 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:31.917129 21027 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:38019
I20260810 03:18:31.917176 21098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:38019 every 8 connection(s)
I20260810 03:18:31.919317 21027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:18:31.924823 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 21027
I20260810 03:18:31.924888 21099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:31.925108 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:18:31.925835 20443 external_mini_cluster.cc:1428] Setting key c7f0d8c5499ce8fe17886ce673815d8c
I20260810 03:18:31.942068 21099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b: Bootstrap starting.
Aug 10 03:18:31 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:31.945518 21099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:31.947046 21099 log.cc:824] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:31.950937 21099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b: No bootstrap required, opened a new log
I20260810 03:18:31.955500 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:31.928825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.19.246.254:38019)
0810 03:18:31.929870 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:31.929883 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:31.929930 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:18:31.933286 (+ 3356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:31.933326 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:31.934032 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:31.934600 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:31.936242 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:31.938372 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:31.939982 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:31.941135 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:31.941329 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:31.945734 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:31.945748 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:31.945771 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:31.945826 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:31.950232 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:31.951088 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:31.951097 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:31.951107 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:31.951342 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:31.951793 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:31.951798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:31.951801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:31.952570 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:31.952878 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:31.953422 (+ 544us) server_negotiation.cc:299] Negotiation successful
0810 03:18:31.953929 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":535,"threads_started":1}
I20260810 03:18:31.961187 21099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd3ec0daf1b45c499e531861db05b8b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38019 } }
I20260810 03:18:31.961645 21099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:31.961760 21099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd3ec0daf1b45c499e531861db05b8b, State: Initialized, Role: FOLLOWER
I20260810 03:18:31.962472 21099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd3ec0daf1b45c499e531861db05b8b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38019 } }
I20260810 03:18:31.962795 21099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:18:31.962953 21099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:18:31.963145 21099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:31.965655 21099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd3ec0daf1b45c499e531861db05b8b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38019 } }
I20260810 03:18:31.966239 21099 leader_election.cc:302] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dd3ec0daf1b45c499e531861db05b8b; no voters:
I20260810 03:18:31.966997 21099 leader_election.cc:288] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:18:31.967192 21104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:31.968653 21104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [term 1 LEADER]: Becoming Leader. State: Replica: 8dd3ec0daf1b45c499e531861db05b8b, State: Running, Role: LEADER
I20260810 03:18:31.969287 21104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd3ec0daf1b45c499e531861db05b8b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38019 } }
I20260810 03:18:31.969663 21099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:18:31.972847 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dd3ec0daf1b45c499e531861db05b8b. Latest consensus state: current_term: 1 leader_uuid: "8dd3ec0daf1b45c499e531861db05b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd3ec0daf1b45c499e531861db05b8b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38019 } } }
I20260810 03:18:31.973402 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:31.973237 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dd3ec0daf1b45c499e531861db05b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd3ec0daf1b45c499e531861db05b8b" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38019 } } }
I20260810 03:18:31.973543 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:31.981700 21109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:18:31.987778 21109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:18:31.998106 21109 catalog_manager.cc:1347] Generated new cluster ID: 5d05e01629cd468a8c1f55b1b0966f8d
I20260810 03:18:31.998257 21109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:18:32.007010 21109 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:18:32.009056 21109 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:18:32.021314 21109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dd3ec0daf1b45c499e531861db05b8b: Generated new TSK 0
I20260810 03:18:32.022574 21109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:18:32.091936 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38019
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:32.447688 21127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:32.448002 21127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:32.448172 21127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:32.456651 21127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:32.456777 21127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:32.456956 21127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:18:32.467442 21127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:32.468958 21127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:32.470605 21127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:32.485244 21135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:32.485241 21133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:32.485596 21132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:32.486079 21127 server_base.cc:1034] running on GCE node
I20260810 03:18:32.486771 21127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:32.487867 21127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:32.489140 21127 hybrid_clock.cc:648] HybridClock initialized: now 1786331912489092 us; error 55 us; skew 500 ppm
I20260810 03:18:32.489537 21127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:32 dist-test-slave-btw8 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331912, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:32.493475 21127 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:18:32.496011 21127 webserver.cc:460] Webserver started at http://127.19.246.193:41965/ using document root <none> and password file <none>
I20260810 03:18:32.496803 21127 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:32.496928 21127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:32.497242 21127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:32.500718 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "a12a15bc9a4f4382817499296f16c76b"
format_stamp: "Formatted at 2026-08-10 03:18:32 on dist-test-slave-btw8"
server_key: "f38113610e602b06b2b781a2f31648aa"
server_key_iv: "e0e9581ac13a1e44ec8e793405a6c457"
server_key_version: "encryptionkey@0"
I20260810 03:18:32.501585 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "a12a15bc9a4f4382817499296f16c76b"
format_stamp: "Formatted at 2026-08-10 03:18:32 on dist-test-slave-btw8"
server_key: "f38113610e602b06b2b781a2f31648aa"
server_key_iv: "e0e9581ac13a1e44ec8e793405a6c457"
server_key_version: "encryptionkey@0"
I20260810 03:18:32.507647 21127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:18:32.511905 21143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:32.513510 21127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:18:32.513727 21127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "a12a15bc9a4f4382817499296f16c76b"
format_stamp: "Formatted at 2026-08-10 03:18:32 on dist-test-slave-btw8"
server_key: "f38113610e602b06b2b781a2f31648aa"
server_key_iv: "e0e9581ac13a1e44ec8e793405a6c457"
server_key_version: "encryptionkey@0"
I20260810 03:18:32.513957 21127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:32.526763 21127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:32.531598 21127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:32.531911 21127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:32.533010 21127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:32.534456 21127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:32.534533 21127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:32.534663 21127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:32.534734 21127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:32.571623 21127 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:44591
I20260810 03:18:32.571673 21263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:44591 every 8 connection(s)
I20260810 03:18:32.573052 21127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:18:32.578249 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 21127
I20260810 03:18:32.578488 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:18:32.579128 20443 external_mini_cluster.cc:1428] Setting key d9ab394b244a012c989dab88d93c6280
Aug 10 03:18:32 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331912, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:32.597926 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:32.575802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:36509 (local address 127.19.246.254:38019)
0810 03:18:32.576268 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:32.576273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:32.577283 (+ 1010us) server_negotiation.cc:407] Connection header received
0810 03:18:32.579381 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:32.579387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:32.579478 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:32.579603 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:32.581531 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:32.582712 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:32.584592 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:32.585379 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:32.585535 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:32.591007 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:32.591015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:32.591020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:32.591079 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:32.593985 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:32.595168 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:32.595173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:32.595176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:32.595308 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:32.595919 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:32.595922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:32.595924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:32.596217 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:32.596328 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:32.597461 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0810 03:18:32.597653 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260810 03:18:32.599733 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:32.576423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:38019 (local address 127.19.246.193:36509)
0810 03:18:32.577110 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:32.577158 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:32.579035 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:32.579861 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:32.579871 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:32.580538 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:32.581261 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:32.581282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:32.582931 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:32.582936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:32.584429 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:32.584442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:32.585673 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:32.585680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:32.585888 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:32.586804 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:32.586836 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:32.590691 (+ 3855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:32.594183 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:32.594240 (+ 57us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:32.594265 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:32.595001 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:32.595491 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:32.595498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:32.595501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:32.595785 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:32.596387 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:32.596399 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:32.596972 (+ 573us) client_negotiation.cc:769] Sending connection context
0810 03:18:32.597334 (+ 362us) client_negotiation.cc:240] Negotiation successful
0810 03:18:32.598048 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
I20260810 03:18:32.601567 21264 heartbeater.cc:347] Connected to a master server at 127.19.246.254:38019
I20260810 03:18:32.601918 21264 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:32.602730 21264 heartbeater.cc:511] Master 127.19.246.254:38019 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
I20260810 03:18:32.605304 21061 ts_manager.cc:194] Registered new tserver with Master: a12a15bc9a4f4382817499296f16c76b (127.19.246.193:44591)
I20260810 03:18:32.606824 21061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:36509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:18:32.631271 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38019
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:32.981959 21271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:32.982326 21271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:32.982436 21271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:32.990350 21271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:32.990489 21271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:32.990609 21271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:18:33.001083 21271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:33.002574 21271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:33.004091 21271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:33.018133 21277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:33.018177 21278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:33.018095 21280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:33.019884 21271 server_base.cc:1034] running on GCE node
I20260810 03:18:33.020753 21271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:33.022090 21271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:33.023415 21271 hybrid_clock.cc:648] HybridClock initialized: now 1786331913023258 us; error 92 us; skew 500 ppm
I20260810 03:18:33.023787 21271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:33 dist-test-slave-btw8 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331913, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:33.027472 21271 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:18:33.029260 21271 webserver.cc:460] Webserver started at http://127.19.246.194:42279/ using document root <none> and password file <none>
I20260810 03:18:33.030036 21271 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:33.030179 21271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:33.030537 21271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:33.034003 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "9e7f3f36bba9466da840d7f3b5eae244"
format_stamp: "Formatted at 2026-08-10 03:18:33 on dist-test-slave-btw8"
server_key: "cd50169185b855bda3362bdc0279c5c7"
server_key_iv: "0610ac77455618ce9a7991d50ace2b0b"
server_key_version: "encryptionkey@0"
I20260810 03:18:33.034874 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "9e7f3f36bba9466da840d7f3b5eae244"
format_stamp: "Formatted at 2026-08-10 03:18:33 on dist-test-slave-btw8"
server_key: "cd50169185b855bda3362bdc0279c5c7"
server_key_iv: "0610ac77455618ce9a7991d50ace2b0b"
server_key_version: "encryptionkey@0"
I20260810 03:18:33.040902 21271 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260810 03:18:33.044910 21287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:33.046536 21271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:18:33.046787 21271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "9e7f3f36bba9466da840d7f3b5eae244"
format_stamp: "Formatted at 2026-08-10 03:18:33 on dist-test-slave-btw8"
server_key: "cd50169185b855bda3362bdc0279c5c7"
server_key_iv: "0610ac77455618ce9a7991d50ace2b0b"
server_key_version: "encryptionkey@0"
I20260810 03:18:33.047025 21271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:33.075069 21271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:33.080206 21271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:33.080546 21271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:33.081758 21271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:33.083462 21271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:33.083573 21271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:33.083774 21271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:33.083855 21271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:33.123307 21271 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:44879
I20260810 03:18:33.123565 21406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:44879 every 8 connection(s)
I20260810 03:18:33.124833 21271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:18:33.135244 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 21271
I20260810 03:18:33.135521 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:18:33.136047 20443 external_mini_cluster.cc:1428] Setting key e77a3cbbaf927f97891c01f62853efed
Aug 10 03:18:33 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331913, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:33.150130 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.127750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:57919 (local address 127.19.246.254:38019)
0810 03:18:33.128273 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:33.128278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:33.129223 (+ 945us) server_negotiation.cc:407] Connection header received
0810 03:18:33.131227 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:33.131248 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:33.131347 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:33.131514 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:33.133185 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.133939 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.135685 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.136192 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.136342 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.141204 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:33.141213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:33.141222 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:33.141273 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:33.145369 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:33.146849 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:33.146854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:33.146856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:33.147030 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:33.147765 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:33.147770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:33.147772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:33.148186 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:33.148341 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:33.149435 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0810 03:18:33.149800 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":297,"threads_started":1}
I20260810 03:18:33.152045 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.128180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:38019 (local address 127.19.246.194:57919)
0810 03:18:33.129009 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:33.129047 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:33.130908 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:33.131802 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:33.131811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:33.132369 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:33.132993 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.133010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.134099 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.134104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.135484 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.135504 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.136458 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.136466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.136805 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.137692 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:33.137724 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:33.140858 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:33.145653 (+ 4795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:33.145663 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:33.145682 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:33.146674 (+ 992us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:33.147209 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:33.147217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:33.147223 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:33.147584 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:33.148339 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:33.148352 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:33.148935 (+ 583us) client_negotiation.cc:769] Sending connection context
0810 03:18:33.149296 (+ 361us) client_negotiation.cc:240] Negotiation successful
0810 03:18:33.150012 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":392,"threads_started":1}
I20260810 03:18:33.153924 21407 heartbeater.cc:347] Connected to a master server at 127.19.246.254:38019
I20260810 03:18:33.154281 21407 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:33.155047 21407 heartbeater.cc:511] Master 127.19.246.254:38019 requested a full tablet report, sending...
I20260810 03:18:33.156840 21061 ts_manager.cc:194] Registered new tserver with Master: 9e7f3f36bba9466da840d7f3b5eae244 (127.19.246.194:44879)
I20260810 03:18:33.157734 21061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:57919
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:18:33.195240 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38019
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:33.545701 21413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:33.546098 21413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:33.546296 21413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:33.554865 21413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:33.555087 21413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:33.555311 21413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:18:33.566340 21413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:33.567893 21413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:33.569363 21413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:33.586190 21418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:33.586609 21419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:33.586974 21422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:33.587194 21413 server_base.cc:1034] running on GCE node
I20260810 03:18:33.587895 21413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:33.588862 21413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:33.590111 21413 hybrid_clock.cc:648] HybridClock initialized: now 1786331913590061 us; error 60 us; skew 500 ppm
I20260810 03:18:33.590592 21413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:33 dist-test-slave-btw8 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331913, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:33.594866 21413 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:18:33.596596 21413 webserver.cc:460] Webserver started at http://127.19.246.195:34477/ using document root <none> and password file <none>
I20260810 03:18:33.597321 21413 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:33.597431 21413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:33.597708 21413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:33.600935 21413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6"
format_stamp: "Formatted at 2026-08-10 03:18:33 on dist-test-slave-btw8"
server_key: "6eed7d369507bc5c415f625db324c7a9"
server_key_iv: "01dd84742a11b31126b17298740de4a4"
server_key_version: "encryptionkey@0"
I20260810 03:18:33.601722 21413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6"
format_stamp: "Formatted at 2026-08-10 03:18:33 on dist-test-slave-btw8"
server_key: "6eed7d369507bc5c415f625db324c7a9"
server_key_iv: "01dd84742a11b31126b17298740de4a4"
server_key_version: "encryptionkey@0"
I20260810 03:18:33.607511 21413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 03:18:33.610306 21264 heartbeater.cc:503] Master 127.19.246.254:38019 was elected leader, sending a full tablet report...
I20260810 03:18:33.611788 21429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:33.614081 21413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:18:33.614323 21413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6"
format_stamp: "Formatted at 2026-08-10 03:18:33 on dist-test-slave-btw8"
server_key: "6eed7d369507bc5c415f625db324c7a9"
server_key_iv: "01dd84742a11b31126b17298740de4a4"
server_key_version: "encryptionkey@0"
I20260810 03:18:33.614567 21413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:33.648591 21413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:33.654417 21413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:33.654791 21413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:33.655982 21413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:33.657651 21413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:33.657769 21413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:33.657939 21413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:33.658015 21413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:33.695005 21413 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:43949
I20260810 03:18:33.695075 21546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:43949 every 8 connection(s)
I20260810 03:18:33.696434 21413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:18:33.698899 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 21413
I20260810 03:18:33.699105 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:18:33.699745 20443 external_mini_cluster.cc:1428] Setting key 44c7571cbf2d96766b754877990eed83
Aug 10 03:18:33 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331913, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:33.720436 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.699338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:52885 (local address 127.19.246.254:38019)
0810 03:18:33.699871 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:33.699878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:33.701071 (+ 1193us) server_negotiation.cc:407] Connection header received
0810 03:18:33.703656 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:33.703661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:33.703775 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:33.703897 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:33.705570 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.706355 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.708113 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.708548 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.708689 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.712763 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:33.712768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:33.712771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:33.712813 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:33.716518 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:33.717446 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:33.717451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:33.717453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:33.717652 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:33.718282 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:33.718318 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:33.718323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:33.718694 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:33.718837 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:33.719796 (+ 959us) server_negotiation.cc:299] Negotiation successful
0810 03:18:33.720038 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260810 03:18:33.721513 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.700002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:38019 (local address 127.19.246.195:52885)
0810 03:18:33.700882 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:33.700949 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:33.703310 (+ 2361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:33.704174 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:33.704183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:33.704768 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:33.705384 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.705402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.706537 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.706542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.707951 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.707964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.708797 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.708804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.709051 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.709851 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:33.709881 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:33.712480 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:33.716675 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:33.716684 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:33.716701 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:33.717286 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:33.717789 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:33.717793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:33.717796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:33.718128 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:33.718827 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:33.718833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:33.719270 (+ 437us) client_negotiation.cc:769] Sending connection context
0810 03:18:33.719657 (+ 387us) client_negotiation.cc:240] Negotiation successful
0810 03:18:33.720249 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":370,"threads_started":1}
I20260810 03:18:33.723075 21547 heartbeater.cc:347] Connected to a master server at 127.19.246.254:38019
I20260810 03:18:33.723479 21547 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:33.724434 21547 heartbeater.cc:511] Master 127.19.246.254:38019 requested a full tablet report, sending...
I20260810 03:18:33.726081 21061 ts_manager.cc:194] Registered new tserver with Master: b83f7cedf58745bba1a0bbd68e8dc9d6 (127.19.246.195:43949)
I20260810 03:18:33.727125 21061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:52885
I20260810 03:18:33.729336 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:18:33.751914 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.737207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.19.246.254:38019)
0810 03:18:33.737405 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:33.737410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:33.737990 (+ 580us) server_negotiation.cc:407] Connection header received
0810 03:18:33.738164 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:33.738168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:33.738252 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:33.738350 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:33.740148 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.740948 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.742353 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.742772 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.742876 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.745562 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:33.745567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:33.745570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:33.745608 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:33.748477 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:33.749420 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:33.749425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:33.749427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:33.749583 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:33.750038 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:33.750053 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:33.750055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:33.750568 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:33.750819 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:33.751290 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 03:18:33.751645 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:18:33.761826 21060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:18:33.770789 21060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:33.799923 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.788416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.19.246.194:44879)
0810 03:18:33.790033 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:33.790039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:33.790098 (+ 59us) server_negotiation.cc:407] Connection header received
0810 03:18:33.790210 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:33.790219 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:33.790523 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:33.790989 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:33.792392 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.794033 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.797243 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.798539 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.798759 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.798960 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:33.799303 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 03:18:33.799537 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":926,"threads_started":1}
I20260810 03:18:33.799993 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.788220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:44879 (local address 127.0.0.1:33976)
0810 03:18:33.789136 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:33.789418 (+ 282us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:33.789686 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:33.790807 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:33.790815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:33.790849 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:33.792156 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.792187 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.794173 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.794180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.797015 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.797033 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.798777 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.798783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.798994 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.799047 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 03:18:33.799236 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 03:18:33.799537 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":339,"threads_started":1}
I20260810 03:18:33.804440 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.789743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.19.246.195:43949)
0810 03:18:33.790600 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:33.790608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:33.794457 (+ 3849us) server_negotiation.cc:407] Connection header received
0810 03:18:33.794839 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:33.794853 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:33.795208 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:33.795479 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:33.796829 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.798839 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.802118 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.803092 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.803323 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.803572 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:33.803854 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 03:18:33.804037 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":897,"threads_started":1}
I20260810 03:18:33.805007 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.790987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.19.246.193:44591)
0810 03:18:33.795520 (+ 4533us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:33.795530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:33.795570 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:18:33.795670 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:33.795678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:33.795904 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:33.796104 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:33.797304 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.799013 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.801778 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.803539 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.803777 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.803936 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:33.804235 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 03:18:33.804450 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4319,"thread_start_us":4089,"threads_started":1}
I20260810 03:18:33.805141 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.790461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:44591 (local address 127.0.0.1:49552)
0810 03:18:33.792926 (+ 2465us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:33.792947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:33.793083 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:33.796135 (+ 3052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:33.796143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:33.796190 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:33.797045 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.797060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.799167 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.799175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.801571 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.801586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.803789 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.803795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.803963 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.803997 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:18:33.804147 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 03:18:33.804353 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"mutex_wait_us":1,"thread_start_us":1610,"threads_started":1}
I20260810 03:18:33.805609 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:33.789529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:43949 (local address 127.0.0.1:34076)
0810 03:18:33.794319 (+ 4790us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:33.794343 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:33.794585 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:33.795480 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:33.795485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:33.795510 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:33.796484 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.796537 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.799024 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.799029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.801937 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:33.801951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:33.803403 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:33.803411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:33.803607 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:33.803649 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:18:33.804880 (+ 1231us) client_negotiation.cc:240] Negotiation successful
0810 03:18:33.805067 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2731,"mutex_wait_us":6,"thread_start_us":2275,"threads_started":1}
I20260810 03:18:33.805259 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 92b25b574b3a4f00a2d30cb5cb31deda (DEFAULT_TABLE table=test-table [id=3ad873eca3dd4dd5b738496d21a76f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:33.807950 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b25b574b3a4f00a2d30cb5cb31deda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:33.809609 21479 tablet_service.cc:1467] Processing CreateTablet for tablet 92b25b574b3a4f00a2d30cb5cb31deda (DEFAULT_TABLE table=test-table [id=3ad873eca3dd4dd5b738496d21a76f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:33.809665 21195 tablet_service.cc:1467] Processing CreateTablet for tablet 92b25b574b3a4f00a2d30cb5cb31deda (DEFAULT_TABLE table=test-table [id=3ad873eca3dd4dd5b738496d21a76f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:33.811210 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b25b574b3a4f00a2d30cb5cb31deda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:33.811225 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b25b574b3a4f00a2d30cb5cb31deda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:33.824425 21567 tablet_bootstrap.cc:492] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244: Bootstrap starting.
I20260810 03:18:33.827596 21568 tablet_bootstrap.cc:492] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b: Bootstrap starting.
I20260810 03:18:33.829193 21570 tablet_bootstrap.cc:492] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6: Bootstrap starting.
I20260810 03:18:33.830262 21567 tablet_bootstrap.cc:654] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:33.832568 21568 tablet_bootstrap.cc:654] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:33.833252 21570 tablet_bootstrap.cc:654] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:33.833230 21567 log.cc:824] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:33.834106 21568 log.cc:824] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:33.834715 21570 log.cc:824] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:33.836952 21567 tablet_bootstrap.cc:492] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244: No bootstrap required, opened a new log
I20260810 03:18:33.837281 21567 ts_tablet_manager.cc:1397] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260810 03:18:33.837997 21570 tablet_bootstrap.cc:492] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6: No bootstrap required, opened a new log
I20260810 03:18:33.838277 21570 ts_tablet_manager.cc:1397] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260810 03:18:33.841513 21568 tablet_bootstrap.cc:492] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b: No bootstrap required, opened a new log
I20260810 03:18:33.841830 21568 ts_tablet_manager.cc:1397] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260810 03:18:33.847113 21567 raft_consensus.cc:348] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:33.847680 21567 raft_consensus.cc:374] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:33.847805 21567 raft_consensus.cc:729] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7f3f36bba9466da840d7f3b5eae244, State: Initialized, Role: FOLLOWER
I20260810 03:18:33.848526 21567 consensus_queue.cc:260] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:33.849989 21407 heartbeater.cc:503] Master 127.19.246.254:38019 was elected leader, sending a full tablet report...
I20260810 03:18:33.849382 21570 raft_consensus.cc:348] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:33.850610 21570 raft_consensus.cc:374] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:33.850798 21570 raft_consensus.cc:729] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83f7cedf58745bba1a0bbd68e8dc9d6, State: Initialized, Role: FOLLOWER
I20260810 03:18:33.850977 21567 ts_tablet_manager.cc:1428] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260810 03:18:33.851737 21570 consensus_queue.cc:260] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:33.853070 21547 heartbeater.cc:503] Master 127.19.246.254:38019 was elected leader, sending a full tablet report...
I20260810 03:18:33.853734 21568 raft_consensus.cc:348] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:33.854039 21570 ts_tablet_manager.cc:1428] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260810 03:18:33.854182 21568 raft_consensus.cc:374] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:33.854282 21568 raft_consensus.cc:729] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12a15bc9a4f4382817499296f16c76b, State: Initialized, Role: FOLLOWER
I20260810 03:18:33.854969 21568 consensus_queue.cc:260] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:33.856915 21568 ts_tablet_manager.cc:1428] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
W20260810 03:18:33.877121 21408 tablet.cc:2369] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:18:33.948407 21548 tablet.cc:2369] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:18:34.058732 21576 raft_consensus.cc:482] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:34.059136 21576 raft_consensus.cc:504] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:34.061504 21576 leader_election.cc:288] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e7f3f36bba9466da840d7f3b5eae244 (127.19.246.194:44879), b83f7cedf58745bba1a0bbd68e8dc9d6 (127.19.246.195:43949)
I20260810 03:18:34.068789 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.061817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:54877 (local address 127.19.246.194:44879)
0810 03:18:34.062060 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:34.062068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:34.063185 (+ 1117us) server_negotiation.cc:407] Connection header received
0810 03:18:34.063468 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:34.063472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:34.063535 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:34.063638 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:34.064232 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.064955 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.066892 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.067780 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.067977 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.068066 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:34.068349 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 03:18:34.068483 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:18:34.068933 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.061705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:44879 (local address 127.19.246.193:54877)
0810 03:18:34.063050 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:34.063071 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:34.063275 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:34.063657 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:34.063662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:34.063687 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:34.064076 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:34.064084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.065109 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.065115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:34.066714 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:34.066728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.068006 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.068012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:34.068182 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.068208 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:18:34.068322 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:18:34.068481 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":4,"thread_start_us":983,"threads_started":1}
I20260810 03:18:34.069310 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.062090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:49673 (local address 127.19.246.195:43949)
0810 03:18:34.062327 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:34.062336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:34.063279 (+ 943us) server_negotiation.cc:407] Connection header received
0810 03:18:34.063511 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:34.063523 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:34.063592 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:34.063711 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:34.064457 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.065259 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.067232 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.068198 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.068410 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.068497 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:34.068884 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 03:18:34.069029 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:18:34.069459 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.062007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:43949 (local address 127.19.246.193:49673)
0810 03:18:34.063165 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:34.063182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:34.063310 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:34.063726 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:34.063731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:34.063763 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:34.064257 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:34.064267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.065427 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.065441 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:34.067044 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:34.067058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.068450 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.068457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:34.068640 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.068665 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:18:34.068793 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 03:18:34.068965 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":726,"threads_started":1}
I20260810 03:18:34.070664 21501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b25b574b3a4f00a2d30cb5cb31deda" candidate_uuid: "a12a15bc9a4f4382817499296f16c76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" is_pre_election: true
I20260810 03:18:34.070590 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b25b574b3a4f00a2d30cb5cb31deda" candidate_uuid: "a12a15bc9a4f4382817499296f16c76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7f3f36bba9466da840d7f3b5eae244" is_pre_election: true
I20260810 03:18:34.071339 21358 raft_consensus.cc:2454] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a12a15bc9a4f4382817499296f16c76b in term 0.
I20260810 03:18:34.071339 21501 raft_consensus.cc:2454] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a12a15bc9a4f4382817499296f16c76b in term 0.
I20260810 03:18:34.074164 21147 leader_election.cc:302] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e7f3f36bba9466da840d7f3b5eae244, a12a15bc9a4f4382817499296f16c76b; no voters:
I20260810 03:18:34.074705 21576 raft_consensus.cc:2781] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:34.074898 21576 raft_consensus.cc:482] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:34.075026 21576 raft_consensus.cc:3037] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 0 FOLLOWER]: Advancing to term 1
W20260810 03:18:34.075944 21266 tablet.cc:2369] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:18:34.077963 21576 raft_consensus.cc:504] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:34.078934 21576 leader_election.cc:288] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [CANDIDATE]: Term 1 election: Requested vote from peers 9e7f3f36bba9466da840d7f3b5eae244 (127.19.246.194:44879), b83f7cedf58745bba1a0bbd68e8dc9d6 (127.19.246.195:43949)
I20260810 03:18:34.079717 21358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b25b574b3a4f00a2d30cb5cb31deda" candidate_uuid: "a12a15bc9a4f4382817499296f16c76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e7f3f36bba9466da840d7f3b5eae244"
I20260810 03:18:34.079717 21501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b25b574b3a4f00a2d30cb5cb31deda" candidate_uuid: "a12a15bc9a4f4382817499296f16c76b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6"
I20260810 03:18:34.079988 21501 raft_consensus.cc:3037] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:34.079985 21358 raft_consensus.cc:3037] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:34.083779 21358 raft_consensus.cc:2454] T 92b25b574b3a4f00a2d30cb5cb31deda P 9e7f3f36bba9466da840d7f3b5eae244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a12a15bc9a4f4382817499296f16c76b in term 1.
I20260810 03:18:34.083779 21501 raft_consensus.cc:2454] T 92b25b574b3a4f00a2d30cb5cb31deda P b83f7cedf58745bba1a0bbd68e8dc9d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a12a15bc9a4f4382817499296f16c76b in term 1.
I20260810 03:18:34.084408 21146 leader_election.cc:302] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a12a15bc9a4f4382817499296f16c76b, b83f7cedf58745bba1a0bbd68e8dc9d6; no voters:
I20260810 03:18:34.084765 21576 raft_consensus.cc:2781] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:34.085323 21576 raft_consensus.cc:686] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [term 1 LEADER]: Becoming Leader. State: Replica: a12a15bc9a4f4382817499296f16c76b, State: Running, Role: LEADER
I20260810 03:18:34.085852 21576 consensus_queue.cc:237] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } }
I20260810 03:18:34.091950 21059 catalog_manager.cc:5495] T 92b25b574b3a4f00a2d30cb5cb31deda P a12a15bc9a4f4382817499296f16c76b reported cstate change: term changed from 0 to 1, leader changed from <none> to a12a15bc9a4f4382817499296f16c76b (127.19.246.193). New cstate: current_term: 1 leader_uuid: "a12a15bc9a4f4382817499296f16c76b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e7f3f36bba9466da840d7f3b5eae244" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 44879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a12a15bc9a4f4382817499296f16c76b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83f7cedf58745bba1a0bbd68e8dc9d6" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43949 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:18:34.181763 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.164520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.19.246.193:44591)
0810 03:18:34.164850 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:34.164860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:34.165323 (+ 463us) server_negotiation.cc:407] Connection header received
0810 03:18:34.165532 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:34.165539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:34.165612 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:34.165751 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:34.167322 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.168145 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.169685 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.170244 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.170407 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.175255 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:34.175274 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:34.175292 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:34.175337 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:34.178661 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.179529 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.179536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.179538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.179720 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.180142 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.180147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.180151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.180540 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:34.180876 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:34.181148 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 03:18:34.181417 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:18:34.237027 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.221921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.19.246.193:44591)
0810 03:18:34.222227 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:34.222237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:34.222650 (+ 413us) server_negotiation.cc:407] Connection header received
0810 03:18:34.222799 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:34.222804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:34.222872 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:34.222962 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:34.224556 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.225342 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.226601 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.227102 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.227243 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.231255 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:34.231260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:34.231264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:34.231405 (+ 141us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:34.234229 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.234965 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.234970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.234972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.235120 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.235682 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.235686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.235687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.235966 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:34.236226 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:34.236472 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 03:18:34.236663 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:18:34.294713 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.278038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.19.246.193:44591)
0810 03:18:34.278324 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:34.278331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:34.278562 (+ 231us) server_negotiation.cc:407] Connection header received
0810 03:18:34.278675 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:34.278679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:34.278766 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:34.278862 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:34.280401 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.281312 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.282850 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.283532 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.283654 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.288329 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:34.288336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:34.288340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:34.288394 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:34.291804 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.292744 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.292750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.292753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.292913 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.293330 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.293334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.293336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.293671 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:34.293892 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:34.294201 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 03:18:34.294382 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260810 03:18:34.297173 21173 tablet_service.cc:2908] Rejecting scan request for tablet 92b25b574b3a4f00a2d30cb5cb31deda: Uninitialized: clean time has not yet been initialized
I20260810 03:18:34.301277 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21127
I20260810 03:18:34.316423 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.302299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.19.246.193:44591)
0810 03:18:34.302494 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:34.302500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:34.302916 (+ 416us) server_negotiation.cc:407] Connection header received
0810 03:18:34.303028 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:34.303032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:34.303101 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:34.303201 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:34.304474 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.305278 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.306566 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.307066 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.307246 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.309893 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:34.309898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:34.309902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:34.309938 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:34.313369 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.314225 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.314231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.314235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.314382 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.314821 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.314826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.314843 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.315146 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:34.315463 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:34.315764 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 03:18:34.316048 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 03:18:34.317309 21259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:34.413885 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 21127
I20260810 03:18:34.436908 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21271
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:18:34.454015 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.437929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.19.246.194:44879)
0810 03:18:34.438111 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:34.438115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:34.438134 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:18:34.438192 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:34.438196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:34.438256 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:34.438341 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:34.439800 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.440532 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.441949 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.442339 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.442438 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.446890 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:34.446908 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:34.446925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:34.446975 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:34.450711 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.451852 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.451860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.451862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.452091 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.452456 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.452459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.452461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.452902 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:34.453132 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:34.453430 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 03:18:34.453610 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:18:34.454736 21402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:34.545496 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 21271
I20260810 03:18:34.565402 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21413
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:18:34.581651 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:34.566221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.19.246.195:43949)
0810 03:18:34.566390 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:34.566395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:34.566415 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:18:34.566641 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:34.566645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:34.566716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:34.566809 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:34.568240 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.569002 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.570583 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:34.570993 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:34.571126 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:34.575560 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:34.575586 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:34.575615 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:34.575664 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:34.578697 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.579508 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.579516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.579519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.579661 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:34.580026 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:34.580029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:34.580031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:34.580417 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:34.580634 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:34.580963 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 03:18:34.581142 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:18:34.582329 21542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:34.675576 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 21413
I20260810 03:18:34.695878 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21027
I20260810 03:18:34.697343 21094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:34.811213 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 21027
2026-08-10T03:18:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4779 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21598](info): set up 2 sockets
Aug 10 03:18:34 dist-test-slave-btw8 krb5kdc[21598](info): commencing operation
krb5kdc: starting...
W20260810 03:18:36.011489 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:18:36 dist-test-slave-btw8 krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:18:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:18:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:18:36.252460 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:42381
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:42381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:36.621898 21611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:36.622334 21611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:36.622558 21611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:36.635152 21611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:18:36.635329 21611 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:18:36.635429 21611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:36.635529 21611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:36.635612 21611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:18:36.635702 21611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:18:36.650008 21611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:42381
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:42381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:36.651559 21611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:36.652974 21611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:36.665395 21616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:36.665483 21619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:36.665429 21617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:36.666918 21611 server_base.cc:1034] running on GCE node
I20260810 03:18:36.667961 21611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:36.669545 21611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:36.670827 21611 hybrid_clock.cc:648] HybridClock initialized: now 1786331916670766 us; error 34 us; skew 500 ppm
I20260810 03:18:36.671252 21611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:36 dist-test-slave-btw8 krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331916, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:36.674729 21611 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:18:36.676426 21611 webserver.cc:460] Webserver started at http://127.19.246.254:40365/ using document root <none> and password file <none>
I20260810 03:18:36.677170 21611 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:36.677265 21611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:36.677547 21611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:36.680730 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "d0495e7719164ec4b336b5042ff12905"
format_stamp: "Formatted at 2026-08-10 03:18:36 on dist-test-slave-btw8"
server_key: "5e1312d9de35c7a3072e459851a07980"
server_key_iv: "c1c7e1fe11aa5b5433d997c9b7fa9a9a"
server_key_version: "encryptionkey@0"
I20260810 03:18:36.681481 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "d0495e7719164ec4b336b5042ff12905"
format_stamp: "Formatted at 2026-08-10 03:18:36 on dist-test-slave-btw8"
server_key: "5e1312d9de35c7a3072e459851a07980"
server_key_iv: "c1c7e1fe11aa5b5433d997c9b7fa9a9a"
server_key_version: "encryptionkey@0"
I20260810 03:18:36.687300 21611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:18:36.691570 21627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:36.693185 21611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:18:36.693408 21611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "d0495e7719164ec4b336b5042ff12905"
format_stamp: "Formatted at 2026-08-10 03:18:36 on dist-test-slave-btw8"
server_key: "5e1312d9de35c7a3072e459851a07980"
server_key_iv: "c1c7e1fe11aa5b5433d997c9b7fa9a9a"
server_key_version: "encryptionkey@0"
I20260810 03:18:36.693728 21611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:36.725641 21611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:36.730654 21611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:36.730983 21611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:36.756055 21611 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:42381
I20260810 03:18:36.756078 21692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:42381 every 8 connection(s)
I20260810 03:18:36.757550 21611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:18:36.759145 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 21611
I20260810 03:18:36.759428 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:18:36.760303 20443 external_mini_cluster.cc:1428] Setting key 743938f3f41fed892d046fb27b8a53aa
I20260810 03:18:36.762974 21693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:18:36 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:36.778527 21693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905: Bootstrap starting.
I20260810 03:18:36.782392 21693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:36.783697 21693 log.cc:824] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:36.785737 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:36.762550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.19.246.254:42381)
0810 03:18:36.763490 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:36.763502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:36.763545 (+ 43us) server_negotiation.cc:407] Connection header received
0810 03:18:36.765215 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:36.765237 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:36.765632 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:36.765979 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:36.767907 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:36.769330 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:36.771545 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:36.772268 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:36.772407 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:36.777858 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:36.777867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:36.777884 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:36.777932 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:36.781254 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:36.782051 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:36.782058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:36.782065 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:36.782237 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:36.782719 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:36.782724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:36.782726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:36.783289 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:36.783657 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:36.784172 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 03:18:36.784666 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":423,"threads_started":1}
I20260810 03:18:36.787060 21693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905: No bootstrap required, opened a new log
I20260810 03:18:36.795342 21693 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0495e7719164ec4b336b5042ff12905" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42381 } }
I20260810 03:18:36.795897 21693 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:36.796017 21693 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0495e7719164ec4b336b5042ff12905, State: Initialized, Role: FOLLOWER
I20260810 03:18:36.796710 21693 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0495e7719164ec4b336b5042ff12905" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42381 } }
I20260810 03:18:36.796995 21693 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:18:36.797170 21693 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:18:36.797422 21693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:36.800235 21693 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0495e7719164ec4b336b5042ff12905" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42381 } }
I20260810 03:18:36.800830 21693 leader_election.cc:302] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0495e7719164ec4b336b5042ff12905; no voters:
I20260810 03:18:36.801556 21693 leader_election.cc:288] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:18:36.801776 21699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:36.803193 21699 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [term 1 LEADER]: Becoming Leader. State: Replica: d0495e7719164ec4b336b5042ff12905, State: Running, Role: LEADER
I20260810 03:18:36.803812 21699 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0495e7719164ec4b336b5042ff12905" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42381 } }
I20260810 03:18:36.804196 21693 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:18:36.806634 21702 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0495e7719164ec4b336b5042ff12905. Latest consensus state: current_term: 1 leader_uuid: "d0495e7719164ec4b336b5042ff12905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0495e7719164ec4b336b5042ff12905" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42381 } } }
I20260810 03:18:36.807054 21701 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0495e7719164ec4b336b5042ff12905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0495e7719164ec4b336b5042ff12905" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42381 } } }
I20260810 03:18:36.807418 21701 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:36.807425 21702 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905 [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:36.809979 21707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:18:36.816064 21707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:18:36.826212 21707 catalog_manager.cc:1347] Generated new cluster ID: 1cce802f93f043ceaa07ae128dd499d3
I20260810 03:18:36.826355 21707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:18:36.839442 21707 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:18:36.841008 21707 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:18:36.848490 21707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905: Generated new TSK 0
I20260810 03:18:36.849747 21707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:18:36.908725 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42381
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:37.269177 21723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:37.269513 21723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:37.269742 21723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:37.277565 21723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:37.277763 21723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:37.277923 21723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:18:37.278072 21723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:18:37.288874 21723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:42381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:37.290310 21723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:37.291850 21723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:37.306457 21729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:37.306496 21728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:37.306420 21731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:37.307391 21723 server_base.cc:1034] running on GCE node
I20260810 03:18:37.308329 21723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:37.309816 21723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:37.311084 21723 hybrid_clock.cc:648] HybridClock initialized: now 1786331917311025 us; error 44 us; skew 500 ppm
I20260810 03:18:37.311524 21723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:37 dist-test-slave-btw8 krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331917, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:37.322355 21723 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:18:37.324293 21723 webserver.cc:460] Webserver started at http://127.19.246.193:41565/ using document root <none> and password file <none>
I20260810 03:18:37.325075 21723 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:37.325222 21723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:37.325579 21723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:37.328743 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "00bc7159923b414191572b41ba1c81f2"
format_stamp: "Formatted at 2026-08-10 03:18:37 on dist-test-slave-btw8"
server_key: "8e0dc9260f837dd6c0d85bb1db4ee552"
server_key_iv: "513fe15236c81c1b1892e55b40b1c893"
server_key_version: "encryptionkey@0"
I20260810 03:18:37.329622 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "00bc7159923b414191572b41ba1c81f2"
format_stamp: "Formatted at 2026-08-10 03:18:37 on dist-test-slave-btw8"
server_key: "8e0dc9260f837dd6c0d85bb1db4ee552"
server_key_iv: "513fe15236c81c1b1892e55b40b1c893"
server_key_version: "encryptionkey@0"
I20260810 03:18:37.335448 21723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:18:37.339637 21738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:37.341113 21723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 03:18:37.341331 21723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "00bc7159923b414191572b41ba1c81f2"
format_stamp: "Formatted at 2026-08-10 03:18:37 on dist-test-slave-btw8"
server_key: "8e0dc9260f837dd6c0d85bb1db4ee552"
server_key_iv: "513fe15236c81c1b1892e55b40b1c893"
server_key_version: "encryptionkey@0"
I20260810 03:18:37.341658 21723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:37.376528 21723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:37.381332 21723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:37.381697 21723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:37.385051 21723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:37.385172 21723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:37.385306 21723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:37.385382 21723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:18:37 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331917, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:37.409729 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:37.385835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:49781 (local address 127.19.246.254:42381)
0810 03:18:37.386296 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:37.386301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:37.387640 (+ 1339us) server_negotiation.cc:407] Connection header received
0810 03:18:37.389776 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:37.389783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:37.389884 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:37.390011 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:37.392463 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.393535 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.396276 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.396857 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.397011 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:37.402806 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:37.402812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:37.402815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:37.402855 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:37.406020 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:37.406865 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:37.406870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:37.406872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:37.406996 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:37.407491 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:37.407502 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:37.407504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:37.407840 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:37.408012 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:37.409205 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0810 03:18:37.409392 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260810 03:18:37.412022 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:37.386548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42381 (local address 127.19.246.193:49781)
0810 03:18:37.387472 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:37.387513 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:37.389465 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:37.390266 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:37.390275 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:37.391104 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:37.392183 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:37.392218 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.393742 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.393750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:37.396028 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:37.396051 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.397144 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.397152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:37.397588 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:37.398611 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:37.398651 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:37.402506 (+ 3855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:37.406194 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:37.406204 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:37.406226 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:37.406715 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:37.407122 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:37.407127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:37.407131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:37.407338 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:37.408006 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:37.408019 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:37.408693 (+ 674us) client_negotiation.cc:769] Sending connection context
0810 03:18:37.409101 (+ 408us) client_negotiation.cc:240] Negotiation successful
0810 03:18:37.409920 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":387,"threads_started":1}
I20260810 03:18:37.435806 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:37.419045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42381 (local address 127.19.246.193:46287)
0810 03:18:37.419886 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:37.419913 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:37.420117 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:37.420510 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:37.420517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:37.421309 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:37.421809 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:37.421826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.423223 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.423228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:37.425117 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:37.425139 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.426269 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.426275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:37.426535 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:37.427483 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:37.427503 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:37.428795 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:37.432061 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:37.432069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:37.432080 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:37.432704 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:37.433277 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:37.433285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:37.433291 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:37.433512 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:37.434215 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:37.434220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:37.434642 (+ 422us) client_negotiation.cc:769] Sending connection context
0810 03:18:37.434826 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 03:18:37.435283 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":446,"threads_started":1}
I20260810 03:18:37.436630 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:37.419035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:46287 (local address 127.19.246.254:42381)
0810 03:18:37.419277 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:37.419286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:37.420042 (+ 756us) server_negotiation.cc:407] Connection header received
0810 03:18:37.420272 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:37.420276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:37.420345 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:37.420434 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:37.422008 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.423004 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.425300 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.425963 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.426148 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:37.429036 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:37.429042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:37.429046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:37.429082 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:37.431890 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:37.432909 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:37.432915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:37.432918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:37.433110 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:37.433653 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:37.433658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:37.433662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:37.434097 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:37.434268 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:37.436176 (+ 1908us) server_negotiation.cc:299] Negotiation successful
0810 03:18:37.436375 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:18:37.447993 21723 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:43649
I20260810 03:18:37.448032 21871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:43649 every 8 connection(s)
I20260810 03:18:37.449311 21723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:18:37.451607 21872 heartbeater.cc:347] Connected to a master server at 127.19.246.254:42381
I20260810 03:18:37.451862 21872 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:37.452564 21872 heartbeater.cc:511] Master 127.19.246.254:42381 requested a full tablet report, sending...
I20260810 03:18:37.453747 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 21723
I20260810 03:18:37.453975 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:18:37.454655 20443 external_mini_cluster.cc:1428] Setting key a427e30c25a957fceaf2719bf164cf78
I20260810 03:18:37.454939 21644 ts_manager.cc:194] Registered new tserver with Master: 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649)
I20260810 03:18:37.456881 21644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:49781
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:18:37.507344 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42381
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:37.854764 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:37.855116 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:37.855300 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:37.862854 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:37.863031 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:37.863201 21876 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:18:37.863346 21876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:18:37.873629 21876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:42381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:37.875077 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:37.876514 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:37.891278 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:37.891542 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:37.891278 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:37.892383 21876 server_base.cc:1034] running on GCE node
I20260810 03:18:37.893108 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:37.894191 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:37.895509 21876 hybrid_clock.cc:648] HybridClock initialized: now 1786331917895447 us; error 47 us; skew 500 ppm
I20260810 03:18:37.895848 21876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:37 dist-test-slave-btw8 krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331917, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:37.899533 21876 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:18:37.905705 21876 webserver.cc:460] Webserver started at http://127.19.246.194:33685/ using document root <none> and password file <none>
I20260810 03:18:37.906783 21876 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:37.906911 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:37.907217 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:37.910475 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "db2c7817be0c4677822f076311b8c825"
format_stamp: "Formatted at 2026-08-10 03:18:37 on dist-test-slave-btw8"
server_key: "f9af8b2a5455cf00ee418b4bfbf4f461"
server_key_iv: "13fc81d1a1445808ac2eb44ffd41aa82"
server_key_version: "encryptionkey@0"
I20260810 03:18:37.911496 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "db2c7817be0c4677822f076311b8c825"
format_stamp: "Formatted at 2026-08-10 03:18:37 on dist-test-slave-btw8"
server_key: "f9af8b2a5455cf00ee418b4bfbf4f461"
server_key_iv: "13fc81d1a1445808ac2eb44ffd41aa82"
server_key_version: "encryptionkey@0"
I20260810 03:18:37.917429 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:18:37.921489 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:37.923147 21876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:18:37.923344 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "db2c7817be0c4677822f076311b8c825"
format_stamp: "Formatted at 2026-08-10 03:18:37 on dist-test-slave-btw8"
server_key: "f9af8b2a5455cf00ee418b4bfbf4f461"
server_key_iv: "13fc81d1a1445808ac2eb44ffd41aa82"
server_key_version: "encryptionkey@0"
I20260810 03:18:37.923605 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:37.965989 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:37.970714 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:37.971035 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:37.974105 21876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:37.974244 21876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:37.974392 21876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:37.974470 21876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:18:37 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331917, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:38.001421 21900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:37.975049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:46509 (local address 127.19.246.254:42381)
0810 03:18:37.975538 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:37.975543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:37.977442 (+ 1899us) server_negotiation.cc:407] Connection header received
0810 03:18:37.979837 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:37.979844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:37.979949 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:37.980037 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:37.982976 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.984424 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.987020 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.987648 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.987801 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:37.993516 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:37.993522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:37.993526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:37.993566 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:37.996656 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:37.998087 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:37.998092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:37.998094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:37.998237 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:37.999074 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:37.999078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:37.999081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:37.999421 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:37.999618 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.000747 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.001013 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":223,"threads_started":1}
I20260810 03:18:38.002723 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:37.976323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42381 (local address 127.19.246.194:46509)
0810 03:18:37.977228 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:37.977281 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:37.979371 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:37.980714 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:37.980722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:37.981663 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:37.982650 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:37.982675 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.984650 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.984657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:37.986840 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:37.986854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:37.988164 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:37.988175 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:37.988527 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:37.989645 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:37.989673 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:37.993234 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:37.996859 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:37.996874 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:37.996911 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:37.997891 (+ 980us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:37.998384 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:37.998393 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:37.998397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:37.998893 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:37.999587 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:37.999596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:38.000230 (+ 634us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.000660 (+ 430us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.001268 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":411,"threads_started":1}
I20260810 03:18:38.029150 21876 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:40475
I20260810 03:18:38.029167 22020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:40475 every 8 connection(s)
I20260810 03:18:38.029634 21900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.009280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:36845 (local address 127.19.246.254:42381)
0810 03:18:38.009541 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.009551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.010204 (+ 653us) server_negotiation.cc:407] Connection header received
0810 03:18:38.010448 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.010452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.010520 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.010609 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:38.013271 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.014652 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.017646 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.018211 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.018338 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.021932 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:38.021942 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:38.021948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:38.022008 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:38.025673 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:38.026835 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:38.026840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:38.026842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:38.026967 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:38.027525 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:38.027532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:38.027537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:38.028053 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:38.028229 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.029040 (+ 811us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.029325 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:18:38.029932 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.009311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42381 (local address 127.19.246.194:36845)
0810 03:18:38.010067 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:38.010094 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:38.010305 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:38.010678 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:38.010685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:38.011590 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:38.012983 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.013009 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.014911 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.014919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.017399 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.017430 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.018464 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.018471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.018673 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.019728 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:38.019777 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:38.021619 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:38.025873 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:38.025883 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:38.025891 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:38.026685 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:38.027094 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:38.027103 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:38.027110 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:38.027342 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:38.028206 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:38.028215 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:38.028554 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.028883 (+ 329us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.029407 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":367,"threads_started":1}
I20260810 03:18:38.031119 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:18:38.033663 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 21876
I20260810 03:18:38.033895 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:18:38.034456 20443 external_mini_cluster.cc:1428] Setting key d385a1007e7fe52ac46ba161d1dede4b
I20260810 03:18:38.035475 22021 heartbeater.cc:347] Connected to a master server at 127.19.246.254:42381
I20260810 03:18:38.035779 22021 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:38.036593 22021 heartbeater.cc:511] Master 127.19.246.254:42381 requested a full tablet report, sending...
I20260810 03:18:38.038143 21644 ts_manager.cc:194] Registered new tserver with Master: db2c7817be0c4677822f076311b8c825 (127.19.246.194:40475)
I20260810 03:18:38.038987 21644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:46509
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:18:38.085348 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42381
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:18:38.460260 21872 heartbeater.cc:503] Master 127.19.246.254:42381 was elected leader, sending a full tablet report...
W20260810 03:18:38.463850 22025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:38.464150 22025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:38.464251 22025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:38.472009 22025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:38.472149 22025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:38.472239 22025 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:18:38.472322 22025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:18:38.482817 22025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:42381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:38.484246 22025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:38.485685 22025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:38.500703 22031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:38.500732 22033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:38.500660 22030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:38.502036 22025 server_base.cc:1034] running on GCE node
I20260810 03:18:38.502836 22025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:38.504002 22025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:38.505286 22025 hybrid_clock.cc:648] HybridClock initialized: now 1786331918505207 us; error 39 us; skew 500 ppm
I20260810 03:18:38.505627 22025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:38 dist-test-slave-btw8 krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331918, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:38.509377 22025 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:18:38.511148 22025 webserver.cc:460] Webserver started at http://127.19.246.195:34079/ using document root <none> and password file <none>
I20260810 03:18:38.511981 22025 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:38.512081 22025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:38.512364 22025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:38.515583 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "0652df4464d54fe7863ddfbf5a14b10e"
format_stamp: "Formatted at 2026-08-10 03:18:38 on dist-test-slave-btw8"
server_key: "08f6748c540fad466701406ca144477e"
server_key_iv: "c93e6b8594e6251d10dbcafd63988293"
server_key_version: "encryptionkey@0"
I20260810 03:18:38.516417 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "0652df4464d54fe7863ddfbf5a14b10e"
format_stamp: "Formatted at 2026-08-10 03:18:38 on dist-test-slave-btw8"
server_key: "08f6748c540fad466701406ca144477e"
server_key_iv: "c93e6b8594e6251d10dbcafd63988293"
server_key_version: "encryptionkey@0"
I20260810 03:18:38.522169 22025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:18:38.526026 22040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:38.527709 22025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:18:38.527863 22025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "0652df4464d54fe7863ddfbf5a14b10e"
format_stamp: "Formatted at 2026-08-10 03:18:38 on dist-test-slave-btw8"
server_key: "08f6748c540fad466701406ca144477e"
server_key_iv: "c93e6b8594e6251d10dbcafd63988293"
server_key_version: "encryptionkey@0"
I20260810 03:18:38.528085 22025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:38.542603 22025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:38.547552 22025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:38.547803 22025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:38.550681 22025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:38.550771 22025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:38.550915 22025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:38.550994 22025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:18:38 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331918, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:38.578933 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.551282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:54115 (local address 127.19.246.254:42381)
0810 03:18:38.552014 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.552022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.553266 (+ 1244us) server_negotiation.cc:407] Connection header received
0810 03:18:38.556602 (+ 3336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.556609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.556771 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.556909 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:38.560006 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.561460 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.564165 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.565009 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.565208 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.571415 (+ 6207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:38.571425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:38.571430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:38.571483 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:38.574893 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:38.576143 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:38.576149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:38.576153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:38.576310 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:38.576836 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:38.576843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:38.576848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:38.577291 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:38.577463 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.578246 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.578504 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":474,"threads_started":1}
I20260810 03:18:38.580552 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.552061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42381 (local address 127.19.246.195:54115)
0810 03:18:38.553036 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:38.553104 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:38.556156 (+ 3052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:38.557312 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:38.557328 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:38.558564 (+ 1236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:38.559706 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.559734 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.561771 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.561778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.563963 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.563979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.565337 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.565349 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.565767 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.566851 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:38.566890 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:38.570935 (+ 4045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:38.575086 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:38.575101 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:38.575127 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:38.575933 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:38.576445 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:38.576451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:38.576454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:38.576691 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:38.577445 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:38.577454 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:38.577857 (+ 403us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.578145 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.578763 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":361,"threads_started":1}
I20260810 03:18:38.606403 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.587107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:46519 (local address 127.19.246.254:42381)
0810 03:18:38.587408 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.587418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.587971 (+ 553us) server_negotiation.cc:407] Connection header received
0810 03:18:38.588200 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.588204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.588272 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.588392 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:38.589889 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.591203 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.593835 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.594697 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.594861 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.597541 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:38.597550 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:38.597559 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:38.597611 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:38.602261 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:38.603660 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:38.603669 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:38.603674 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:38.603933 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:38.604412 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:38.604419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:38.604423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:38.604884 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:38.605066 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.605585 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.605897 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:18:38.606611 22025 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:41615
I20260810 03:18:38.606667 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.587006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42381 (local address 127.19.246.195:46519)
0810 03:18:38.587817 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:38.587851 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:38.588061 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:38.588427 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:38.588432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:38.589211 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:38.589697 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.589719 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.591420 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.591427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.593623 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.593647 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.594978 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.594990 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.595238 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.596269 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:38.596302 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:38.597246 (+ 944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:38.602491 (+ 5245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:38.602501 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:38.602509 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:38.603466 (+ 957us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:38.604062 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:38.604066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:38.604069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:38.604280 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:38.605016 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:38.605020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:38.605271 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.605443 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.605977 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":410,"threads_started":1}
I20260810 03:18:38.607141 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:41615 every 8 connection(s)
I20260810 03:18:38.608304 22025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:18:38.610603 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 22025
I20260810 03:18:38.610847 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:18:38.611485 20443 external_mini_cluster.cc:1428] Setting key 22dc5ea67e25876c4d2b6a468b6e6d54
I20260810 03:18:38.612630 22168 heartbeater.cc:347] Connected to a master server at 127.19.246.254:42381
I20260810 03:18:38.612886 22168 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:38.613871 22168 heartbeater.cc:511] Master 127.19.246.254:42381 requested a full tablet report, sending...
I20260810 03:18:38.615563 21644 ts_manager.cc:194] Registered new tserver with Master: 0652df4464d54fe7863ddfbf5a14b10e (127.19.246.195:41615)
I20260810 03:18:38.616540 21644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:54115
I20260810 03:18:38.617270 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:18:38 dist-test-slave-btw8 krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:18:38 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:38.675509 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.658852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.19.246.254:42381)
0810 03:18:38.659148 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.659156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.659499 (+ 343us) server_negotiation.cc:407] Connection header received
0810 03:18:38.659827 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.659832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.659907 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.660033 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:38.661751 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.662622 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.664190 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.664667 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.664829 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.669403 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:38.669409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:38.669412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:38.669467 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:38.672354 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:38.673141 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:38.673146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:38.673149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:38.673291 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:38.673753 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:38.673758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:38.673760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:38.674065 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:38.674403 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.674726 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.674992 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 03:18:38.680459 21644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:18:38.688117 21644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:38.716145 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.706094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:40475 (local address 127.0.0.1:39520)
0810 03:18:38.707748 (+ 1654us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:38.707766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:38.707891 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:38.709098 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:38.709102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:38.709125 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:38.709447 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.709456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.711284 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.711293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.713128 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.713142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.714848 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.714855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.715103 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.715170 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.715305 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.715640 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1519,"mutex_wait_us":7,"thread_start_us":719,"threads_started":1}
I20260810 03:18:38.716777 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.701821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:41615 (local address 127.0.0.1:37412)
0810 03:18:38.703843 (+ 2022us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:38.703896 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:38.704125 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:38.707596 (+ 3471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:38.707612 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:38.707655 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:38.708323 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.708337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.710595 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.710601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.713781 (+ 3180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.713796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.715867 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.715873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.716077 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.716116 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.716221 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.716374 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1672,"mutex_wait_us":308,"thread_start_us":5979,"threads_started":1}
I20260810 03:18:38.716857 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.702172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.19.246.195:41615)
0810 03:18:38.704994 (+ 2822us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.705007 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.705051 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:18:38.706471 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.706483 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.706828 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.707079 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:38.708556 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.710420 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.713981 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.715637 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.715849 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.716015 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.716327 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.716503 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2596,"thread_start_us":2398,"threads_started":1}
I20260810 03:18:38.716985 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.702775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.19.246.193:43649)
0810 03:18:38.706183 (+ 3408us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.706192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.706973 (+ 781us) server_negotiation.cc:407] Connection header received
0810 03:18:38.707308 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.707316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.707584 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.707758 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:38.708833 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.710419 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.713991 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.715517 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.715731 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.715943 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.716475 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.716655 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3244,"thread_start_us":3118,"threads_started":1}
I20260810 03:18:38.717533 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.702171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:43649 (local address 127.0.0.1:37502)
0810 03:18:38.706832 (+ 4661us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:38.706855 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:38.707102 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:38.708063 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:38.708082 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:38.708109 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:38.708652 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.708664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.710595 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.710601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.713781 (+ 3180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.713796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.716099 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.716105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.716282 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.716312 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.717177 (+ 865us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.717282 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4524,"mutex_wait_us":194,"thread_start_us":1126,"threads_started":1}
I20260810 03:18:38.719281 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.706335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.19.246.194:40475)
0810 03:18:38.708514 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.708525 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.708557 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:18:38.708667 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.708674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.708896 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.709089 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:38.709631 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.711135 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.713305 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.714586 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.718496 (+ 3910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.718711 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.718783 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.718977 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":1800,"threads_started":1}
I20260810 03:18:38.721029 21794 tablet_service.cc:1467] Processing CreateTablet for tablet 06d4888a86b34b5dae4727300d36f225 (DEFAULT_TABLE table=test-table [id=ddeca52e16154ca29445c6bb966615a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:38.721045 22094 tablet_service.cc:1467] Processing CreateTablet for tablet 06d4888a86b34b5dae4727300d36f225 (DEFAULT_TABLE table=test-table [id=ddeca52e16154ca29445c6bb966615a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:38.722338 21794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d4888a86b34b5dae4727300d36f225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:38.722519 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d4888a86b34b5dae4727300d36f225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:38.722841 21945 tablet_service.cc:1467] Processing CreateTablet for tablet 06d4888a86b34b5dae4727300d36f225 (DEFAULT_TABLE table=test-table [id=ddeca52e16154ca29445c6bb966615a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:38.724313 21945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d4888a86b34b5dae4727300d36f225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:38.735868 22190 tablet_bootstrap.cc:492] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: Bootstrap starting.
I20260810 03:18:38.739187 22191 tablet_bootstrap.cc:492] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Bootstrap starting.
I20260810 03:18:38.739866 22190 tablet_bootstrap.cc:654] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:38.741017 22192 tablet_bootstrap.cc:492] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Bootstrap starting.
I20260810 03:18:38.741472 22190 log.cc:824] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:38.743136 22191 tablet_bootstrap.cc:654] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:38.744410 22191 log.cc:824] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:38.744786 22192 tablet_bootstrap.cc:654] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:38.745252 22190 tablet_bootstrap.cc:492] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: No bootstrap required, opened a new log
I20260810 03:18:38.745565 22190 ts_tablet_manager.cc:1397] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260810 03:18:38.746109 22192 log.cc:824] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:38.748003 22191 tablet_bootstrap.cc:492] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: No bootstrap required, opened a new log
I20260810 03:18:38.748332 22191 ts_tablet_manager.cc:1397] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 03:18:38.749421 22192 tablet_bootstrap.cc:492] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: No bootstrap required, opened a new log
I20260810 03:18:38.749716 22192 ts_tablet_manager.cc:1397] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 03:18:38.756865 22191 raft_consensus.cc:348] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.756839 22190 raft_consensus.cc:348] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.757350 22191 raft_consensus.cc:374] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:38.757354 22190 raft_consensus.cc:374] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:38.757439 22191 raft_consensus.cc:729] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0652df4464d54fe7863ddfbf5a14b10e, State: Initialized, Role: FOLLOWER
I20260810 03:18:38.757464 22190 raft_consensus.cc:729] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bc7159923b414191572b41ba1c81f2, State: Initialized, Role: FOLLOWER
I20260810 03:18:38.758082 22191 consensus_queue.cc:260] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.758210 22190 consensus_queue.cc:260] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.761835 22192 raft_consensus.cc:348] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.762512 22192 raft_consensus.cc:374] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:38.762642 22192 raft_consensus.cc:729] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2c7817be0c4677822f076311b8c825, State: Initialized, Role: FOLLOWER
I20260810 03:18:38.763674 22192 consensus_queue.cc:260] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.765704 22168 heartbeater.cc:503] Master 127.19.246.254:42381 was elected leader, sending a full tablet report...
I20260810 03:18:38.767212 22190 ts_tablet_manager.cc:1428] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: Time spent starting tablet: real 0.021s user 0.013s sys 0.010s
I20260810 03:18:38.769707 22192 ts_tablet_manager.cc:1428] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
I20260810 03:18:38.769625 22199 raft_consensus.cc:482] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:38.770176 22021 heartbeater.cc:503] Master 127.19.246.254:42381 was elected leader, sending a full tablet report...
I20260810 03:18:38.769979 22199 raft_consensus.cc:504] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.771783 22191 ts_tablet_manager.cc:1428] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
I20260810 03:18:38.773970 22199 leader_election.cc:288] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0652df4464d54fe7863ddfbf5a14b10e (127.19.246.195:41615), 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649)
W20260810 03:18:38.784089 22022 tablet.cc:2369] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:18:38.784368 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.774510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:34173 (local address 127.19.246.195:41615)
0810 03:18:38.774695 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.774701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.777272 (+ 2571us) server_negotiation.cc:407] Connection header received
0810 03:18:38.777575 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.777582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.777678 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.777773 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:38.778388 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.779292 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.781892 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.783036 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.783226 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.783306 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.783942 (+ 636us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.784084 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:18:38.784569 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.775191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:38259 (local address 127.19.246.193:43649)
0810 03:18:38.775430 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.775436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.777692 (+ 2256us) server_negotiation.cc:407] Connection header received
0810 03:18:38.777884 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.777890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.777965 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.778028 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:38.778813 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.780010 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.782384 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.783484 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.783677 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.783765 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.784176 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.784333 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:18:38.785230 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.774294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:41615 (local address 127.19.246.194:34173)
0810 03:18:38.777157 (+ 2863us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:38.777179 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:38.777416 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:38.777785 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:38.777790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:38.777818 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:38.778201 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.778214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.779535 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.779542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.781695 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.781715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.783502 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.783509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.783737 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.783778 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.784628 (+ 850us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.784784 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2705,"thread_start_us":2941,"threads_started":1}
I20260810 03:18:38.785475 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.774558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:43649 (local address 127.19.246.194:38259)
0810 03:18:38.777550 (+ 2992us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:38.777581 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:38.777710 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:38.778193 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:38.778196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:38.778214 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:38.778640 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.778652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.780172 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.780177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.782236 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:38.782247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.783775 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.783783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:38.783952 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.783985 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:18:38.784831 (+ 846us) client_negotiation.cc:240] Negotiation successful
0810 03:18:38.784999 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2855,"thread_start_us":2228,"threads_started":1}
I20260810 03:18:38.786479 22121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d4888a86b34b5dae4727300d36f225" candidate_uuid: "db2c7817be0c4677822f076311b8c825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0652df4464d54fe7863ddfbf5a14b10e" is_pre_election: true
I20260810 03:18:38.786480 21820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d4888a86b34b5dae4727300d36f225" candidate_uuid: "db2c7817be0c4677822f076311b8c825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bc7159923b414191572b41ba1c81f2" is_pre_election: true
I20260810 03:18:38.787101 21820 raft_consensus.cc:2454] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db2c7817be0c4677822f076311b8c825 in term 0.
I20260810 03:18:38.787107 22121 raft_consensus.cc:2454] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db2c7817be0c4677822f076311b8c825 in term 0.
I20260810 03:18:38.787946 21893 leader_election.cc:302] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00bc7159923b414191572b41ba1c81f2, 0652df4464d54fe7863ddfbf5a14b10e, db2c7817be0c4677822f076311b8c825; no voters:
I20260810 03:18:38.788432 22199 raft_consensus.cc:2781] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:38.788595 22199 raft_consensus.cc:482] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:38.788700 22199 raft_consensus.cc:3037] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:38.790946 22199 raft_consensus.cc:504] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.791697 22199 leader_election.cc:288] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [CANDIDATE]: Term 1 election: Requested vote from peers 0652df4464d54fe7863ddfbf5a14b10e (127.19.246.195:41615), 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649)
I20260810 03:18:38.792198 22121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d4888a86b34b5dae4727300d36f225" candidate_uuid: "db2c7817be0c4677822f076311b8c825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0652df4464d54fe7863ddfbf5a14b10e"
I20260810 03:18:38.792234 21820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d4888a86b34b5dae4727300d36f225" candidate_uuid: "db2c7817be0c4677822f076311b8c825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bc7159923b414191572b41ba1c81f2"
I20260810 03:18:38.792433 22121 raft_consensus.cc:3037] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:38.792446 21820 raft_consensus.cc:3037] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:38.794797 21820 raft_consensus.cc:2454] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db2c7817be0c4677822f076311b8c825 in term 1.
I20260810 03:18:38.794859 22121 raft_consensus.cc:2454] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db2c7817be0c4677822f076311b8c825 in term 1.
I20260810 03:18:38.795323 21893 leader_election.cc:302] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bc7159923b414191572b41ba1c81f2, db2c7817be0c4677822f076311b8c825; no voters:
I20260810 03:18:38.795732 22199 raft_consensus.cc:2781] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:38.796227 22199 raft_consensus.cc:686] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 1 LEADER]: Becoming Leader. State: Replica: db2c7817be0c4677822f076311b8c825, State: Running, Role: LEADER
I20260810 03:18:38.796728 22199 consensus_queue.cc:237] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } }
I20260810 03:18:38.802016 21644 catalog_manager.cc:5495] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 reported cstate change: term changed from 0 to 1, leader changed from <none> to db2c7817be0c4677822f076311b8c825 (127.19.246.194). New cstate: current_term: 1 leader_uuid: "db2c7817be0c4677822f076311b8c825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } health_report { overall_health: HEALTHY } } }
I20260810 03:18:38.858161 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:38.851644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.19.246.194:40475)
0810 03:18:38.851894 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:38.851900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:38.852116 (+ 216us) server_negotiation.cc:407] Connection header received
0810 03:18:38.852531 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:38.852535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:38.852638 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:38.852805 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:38.853202 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.854143 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.855630 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:38.856055 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:38.856195 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:38.856834 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:38.857477 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:38.857664 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:38.857781 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 03:18:38.857871 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 03:18:38.860908 22171 tablet.cc:2369] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:18:38.869623 21820 raft_consensus.cc:1261] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Refusing update from remote peer db2c7817be0c4677822f076311b8c825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:38.869627 22121 raft_consensus.cc:1261] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Refusing update from remote peer db2c7817be0c4677822f076311b8c825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:38.870738 22202 consensus_queue.cc:1035] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:38.871160 22199 consensus_queue.cc:1035] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:38.884550 22206 mvcc.cc:205] Tried to move back new op lower bound from 7316815539671883776 to 7316815539395686400. Current Snapshot: MvccSnapshot[applied={T|T < 7316815539671883776}]
I20260810 03:18:38.886991 22208 mvcc.cc:205] Tried to move back new op lower bound from 7316815539671883776 to 7316815539395686400. Current Snapshot: MvccSnapshot[applied={T|T < 7316815539671883776}]
I20260810 03:18:38.910944 21644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:38.911319 21644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:38.916100 21644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905: Sending DeleteTablet for 3 replicas of tablet 06d4888a86b34b5dae4727300d36f225
I20260810 03:18:38.917950 21945 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d4888a86b34b5dae4727300d36f225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:38 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:39520
I20260810 03:18:38.919803 22094 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d4888a86b34b5dae4727300d36f225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:38 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:37412
I20260810 03:18:38.919806 21794 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d4888a86b34b5dae4727300d36f225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:38 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:37502
I20260810 03:18:38.922694 22217 tablet_replica.cc:330] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: stopping tablet replica
I20260810 03:18:38.922300 21642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:18:38.923439 22217 raft_consensus.cc:2229] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 03:18:38.923765 21642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:38.924058 22217 raft_consensus.cc:2258] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:38.926571 22216 tablet_replica.cc:330] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: stopping tablet replica
I20260810 03:18:38.927242 22216 raft_consensus.cc:2229] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:18:38.928462 22216 raft_consensus.cc:2258] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:38.931321 22216 ts_tablet_manager.cc:1905] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:38.933466 22218 tablet_replica.cc:330] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: stopping tablet replica
I20260810 03:18:38.933962 22218 raft_consensus.cc:2229] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:38.934504 22218 raft_consensus.cc:2258] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:38.937443 21794 tablet_service.cc:1467] Processing CreateTablet for tablet a93283c33cf04c77840891c9e09d337c (DEFAULT_TABLE table=test-table [id=ce5f47bbb6664d1f9511e21946fc8717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:38.937716 22094 tablet_service.cc:1467] Processing CreateTablet for tablet a93283c33cf04c77840891c9e09d337c (DEFAULT_TABLE table=test-table [id=ce5f47bbb6664d1f9511e21946fc8717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:38.937929 21794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93283c33cf04c77840891c9e09d337c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:38.938206 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93283c33cf04c77840891c9e09d337c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:38.938277 21945 tablet_service.cc:1467] Processing CreateTablet for tablet a93283c33cf04c77840891c9e09d337c (DEFAULT_TABLE table=test-table [id=ce5f47bbb6664d1f9511e21946fc8717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:38.938772 21945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93283c33cf04c77840891c9e09d337c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:38.940745 22217 ts_tablet_manager.cc:1905] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:38.944708 22190 tablet_bootstrap.cc:492] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: Bootstrap starting.
I20260810 03:18:38.945178 22192 tablet_bootstrap.cc:492] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: Bootstrap starting.
I20260810 03:18:38.946496 22190 tablet_bootstrap.cc:654] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:38.947031 22192 tablet_bootstrap.cc:654] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:38.948534 22191 tablet_bootstrap.cc:492] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: Bootstrap starting.
I20260810 03:18:38.949185 22216 ts_tablet_manager.cc:1918] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:18:38.949543 22216 log.cc:1196] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/06d4888a86b34b5dae4727300d36f225
I20260810 03:18:38.950315 22216 ts_tablet_manager.cc:1939] T 06d4888a86b34b5dae4727300d36f225 P db2c7817be0c4677822f076311b8c825: Deleting consensus metadata
I20260810 03:18:38.951056 22191 tablet_bootstrap.cc:654] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:38.952589 21631 catalog_manager.cc:4841] TS db2c7817be0c4677822f076311b8c825 (127.19.246.194:40475): tablet 06d4888a86b34b5dae4727300d36f225 (table test-table [id=ddeca52e16154ca29445c6bb966615a9]) successfully deleted
I20260810 03:18:38.952975 22217 ts_tablet_manager.cc:1918] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:18:38.953162 22217 log.cc:1196] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/06d4888a86b34b5dae4727300d36f225
I20260810 03:18:38.953890 22217 ts_tablet_manager.cc:1939] T 06d4888a86b34b5dae4727300d36f225 P 0652df4464d54fe7863ddfbf5a14b10e: Deleting consensus metadata
I20260810 03:18:38.955931 22190 tablet_bootstrap.cc:492] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: No bootstrap required, opened a new log
I20260810 03:18:38.956133 22190 ts_tablet_manager.cc:1397] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260810 03:18:38.956454 21628 catalog_manager.cc:4841] TS 0652df4464d54fe7863ddfbf5a14b10e (127.19.246.195:41615): tablet 06d4888a86b34b5dae4727300d36f225 (table test-table [id=ddeca52e16154ca29445c6bb966615a9]) successfully deleted
I20260810 03:18:38.957168 22190 raft_consensus.cc:348] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:38.957489 22190 raft_consensus.cc:374] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:38.957602 22190 raft_consensus.cc:729] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bc7159923b414191572b41ba1c81f2, State: Initialized, Role: FOLLOWER
I20260810 03:18:38.958001 22190 consensus_queue.cc:260] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:38.958606 22190 ts_tablet_manager.cc:1428] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:18:38.937006 22218 ts_tablet_manager.cc:1905] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:38.968674 22218 ts_tablet_manager.cc:1918] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:18:38.968901 22218 log.cc:1196] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/06d4888a86b34b5dae4727300d36f225
I20260810 03:18:38.969148 22191 tablet_bootstrap.cc:492] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: No bootstrap required, opened a new log
I20260810 03:18:38.969341 22191 ts_tablet_manager.cc:1397] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.004s
I20260810 03:18:38.969509 22218 ts_tablet_manager.cc:1939] T 06d4888a86b34b5dae4727300d36f225 P 00bc7159923b414191572b41ba1c81f2: Deleting consensus metadata
I20260810 03:18:38.970326 22191 raft_consensus.cc:348] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:38.970573 22191 raft_consensus.cc:374] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:38.970675 22191 raft_consensus.cc:729] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0652df4464d54fe7863ddfbf5a14b10e, State: Initialized, Role: FOLLOWER
I20260810 03:18:38.970769 22192 tablet_bootstrap.cc:492] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: No bootstrap required, opened a new log
I20260810 03:18:38.970957 22192 ts_tablet_manager.cc:1397] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.004s
I20260810 03:18:38.971047 22191 consensus_queue.cc:260] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:38.971676 21629 catalog_manager.cc:4841] TS 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649): tablet 06d4888a86b34b5dae4727300d36f225 (table test-table [id=ddeca52e16154ca29445c6bb966615a9]) successfully deleted
I20260810 03:18:38.971958 22192 raft_consensus.cc:348] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:38.972220 22192 raft_consensus.cc:374] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:38.972321 22192 raft_consensus.cc:729] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2c7817be0c4677822f076311b8c825, State: Initialized, Role: FOLLOWER
I20260810 03:18:38.972498 22191 ts_tablet_manager.cc:1428] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 03:18:38.972595 22192 consensus_queue.cc:260] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:38.973389 22192 ts_tablet_manager.cc:1428] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260810 03:18:39.202308 21873 tablet.cc:2369] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:18:39.260149 22196 raft_consensus.cc:482] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:39.260447 22196 raft_consensus.cc:504] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.262177 22196 leader_election.cc:288] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649), db2c7817be0c4677822f076311b8c825 (127.19.246.194:40475)
I20260810 03:18:39.270895 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.262654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:50489 (local address 127.19.246.193:43649)
0810 03:18:39.262903 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.262912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.263683 (+ 771us) server_negotiation.cc:407] Connection header received
0810 03:18:39.264027 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.264035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.264142 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.264256 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:39.264953 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.265876 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.268476 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.269793 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.269940 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.269999 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.270477 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.270624 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:18:39.271075 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.262399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:43649 (local address 127.19.246.195:50489)
0810 03:18:39.263499 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:39.263531 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:39.263814 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:39.264463 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:39.264469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:39.264496 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:39.264780 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.264789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.266022 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.266031 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.268277 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.268293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.270164 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.270168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.270301 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.270325 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:18:39.270447 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 03:18:39.270598 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":6,"spinlock_wait_cycles":896,"thread_start_us":1248,"threads_started":1}
I20260810 03:18:39.271941 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.263020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:34689 (local address 127.19.246.194:40475)
0810 03:18:39.263225 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.263231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.264060 (+ 829us) server_negotiation.cc:407] Connection header received
0810 03:18:39.264245 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.264250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.264355 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.264456 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:39.265277 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.266480 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.268884 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.270353 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.270545 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.270625 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.271387 (+ 762us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.271557 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:18:39.272025 21820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93283c33cf04c77840891c9e09d337c" candidate_uuid: "0652df4464d54fe7863ddfbf5a14b10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bc7159923b414191572b41ba1c81f2" is_pre_election: true
I20260810 03:18:39.272336 21820 raft_consensus.cc:2454] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0652df4464d54fe7863ddfbf5a14b10e in term 0.
I20260810 03:18:39.272727 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.262510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:40475 (local address 127.19.246.195:34689)
0810 03:18:39.263927 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:39.263946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:39.264088 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:39.264502 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:39.264508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:39.264574 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:39.265047 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.265061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.266640 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.266644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.268682 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.268696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.270842 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.270850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.271100 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.271135 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:18:39.271862 (+ 727us) client_negotiation.cc:240] Negotiation successful
0810 03:18:39.272323 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":27,"spinlock_wait_cycles":1792,"thread_start_us":756,"threads_started":1}
I20260810 03:18:39.272981 22042 leader_election.cc:302] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bc7159923b414191572b41ba1c81f2, 0652df4464d54fe7863ddfbf5a14b10e; no voters:
I20260810 03:18:39.273656 22196 raft_consensus.cc:2781] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:39.273797 22196 raft_consensus.cc:482] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:39.273933 22196 raft_consensus.cc:3037] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:39.273981 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93283c33cf04c77840891c9e09d337c" candidate_uuid: "0652df4464d54fe7863ddfbf5a14b10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2c7817be0c4677822f076311b8c825" is_pre_election: true
I20260810 03:18:39.274704 21971 raft_consensus.cc:2454] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0652df4464d54fe7863ddfbf5a14b10e in term 0.
I20260810 03:18:39.277194 22196 raft_consensus.cc:504] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.278155 22196 leader_election.cc:288] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [CANDIDATE]: Term 1 election: Requested vote from peers 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649), db2c7817be0c4677822f076311b8c825 (127.19.246.194:40475)
I20260810 03:18:39.278955 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93283c33cf04c77840891c9e09d337c" candidate_uuid: "0652df4464d54fe7863ddfbf5a14b10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2c7817be0c4677822f076311b8c825"
I20260810 03:18:39.278951 21820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93283c33cf04c77840891c9e09d337c" candidate_uuid: "0652df4464d54fe7863ddfbf5a14b10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bc7159923b414191572b41ba1c81f2"
I20260810 03:18:39.279290 21820 raft_consensus.cc:3037] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:39.279393 21971 raft_consensus.cc:3037] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:39.282907 21820 raft_consensus.cc:2454] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0652df4464d54fe7863ddfbf5a14b10e in term 1.
I20260810 03:18:39.283030 21971 raft_consensus.cc:2454] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0652df4464d54fe7863ddfbf5a14b10e in term 1.
I20260810 03:18:39.283599 22042 leader_election.cc:302] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bc7159923b414191572b41ba1c81f2, 0652df4464d54fe7863ddfbf5a14b10e; no voters:
I20260810 03:18:39.284121 22196 raft_consensus.cc:2781] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:39.284873 22196 raft_consensus.cc:686] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 1 LEADER]: Becoming Leader. State: Replica: 0652df4464d54fe7863ddfbf5a14b10e, State: Running, Role: LEADER
I20260810 03:18:39.285369 22196 consensus_queue.cc:237] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.289767 21642 catalog_manager.cc:5495] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0652df4464d54fe7863ddfbf5a14b10e (127.19.246.195). New cstate: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } health_report { overall_health: HEALTHY } } }
Aug 10 03:18:39 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331916, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:39.364044 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.346661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42381 (local address 127.0.0.1:57156)
0810 03:18:39.348020 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:39.348041 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:39.348228 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:39.348641 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:39.348646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:39.349547 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:39.349816 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.349828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.351002 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.351008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.352668 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.352685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.353531 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.353538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.353718 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.354527 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:39.354561 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:39.357372 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:39.360973 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:39.360993 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:39.361002 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:39.361465 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:39.361949 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:39.361955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:39.361959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:39.362193 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:39.362963 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:39.362973 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:39.363250 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 03:18:39.363474 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 03:18:39.363738 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":1039,"threads_started":1}
I20260810 03:18:39.364424 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.346635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57156 (local address 127.19.246.254:42381)
0810 03:18:39.347707 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.347717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.348174 (+ 457us) server_negotiation.cc:407] Connection header received
0810 03:18:39.348378 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.348383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.348451 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.348579 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:39.350014 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.350837 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.352848 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.353339 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.353446 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.357601 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:39.357607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:39.357610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:39.357649 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:39.360792 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.361602 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.361608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.361611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.361755 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.362348 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.362352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.362355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.362743 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:39.362896 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.363657 (+ 761us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.363930 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":576,"threads_started":1}
I20260810 03:18:39.370877 21642 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:57156:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:18:39.372571 21642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:39.381562 21794 tablet_service.cc:1467] Processing CreateTablet for tablet 1221c129403949b19420833ece16e685 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=527e46d9b4dc42978aa06a20ee4b98b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:18:39.381422 21945 tablet_service.cc:1467] Processing CreateTablet for tablet 1221c129403949b19420833ece16e685 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=527e46d9b4dc42978aa06a20ee4b98b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:18:39.382217 21945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1221c129403949b19420833ece16e685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:39.382254 21794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1221c129403949b19420833ece16e685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:39.386109 22094 tablet_service.cc:1467] Processing CreateTablet for tablet 1221c129403949b19420833ece16e685 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=527e46d9b4dc42978aa06a20ee4b98b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:18:39.386755 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1221c129403949b19420833ece16e685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:39.391001 22190 tablet_bootstrap.cc:492] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: Bootstrap starting.
I20260810 03:18:39.392161 22192 tablet_bootstrap.cc:492] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: Bootstrap starting.
I20260810 03:18:39.392967 22190 tablet_bootstrap.cc:654] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:39.393978 22192 tablet_bootstrap.cc:654] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:39.397696 22191 tablet_bootstrap.cc:492] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: Bootstrap starting.
I20260810 03:18:39.400069 22191 tablet_bootstrap.cc:654] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:39.400508 22192 tablet_bootstrap.cc:492] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: No bootstrap required, opened a new log
I20260810 03:18:39.400774 22192 ts_tablet_manager.cc:1397] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260810 03:18:39.402611 22190 tablet_bootstrap.cc:492] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: No bootstrap required, opened a new log
I20260810 03:18:39.402552 22192 raft_consensus.cc:348] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.402802 22190 ts_tablet_manager.cc:1397] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260810 03:18:39.402834 22192 raft_consensus.cc:374] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:39.402930 22192 raft_consensus.cc:729] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2c7817be0c4677822f076311b8c825, State: Initialized, Role: FOLLOWER
I20260810 03:18:39.403291 22192 consensus_queue.cc:260] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.403673 22190 raft_consensus.cc:348] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.403919 22190 raft_consensus.cc:374] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:39.404002 22190 raft_consensus.cc:729] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bc7159923b414191572b41ba1c81f2, State: Initialized, Role: FOLLOWER
I20260810 03:18:39.403896 22199 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.404165 22199 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:39.404256 22190 consensus_queue.cc:260] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.404989 22190 ts_tablet_manager.cc:1428] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:18:39.405048 22192 ts_tablet_manager.cc:1428] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 03:18:39.406072 22191 tablet_bootstrap.cc:492] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: No bootstrap required, opened a new log
I20260810 03:18:39.406267 22191 ts_tablet_manager.cc:1397] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260810 03:18:39.406139 22197 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.406540 22197 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:39.407001 22191 raft_consensus.cc:348] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.407253 22191 raft_consensus.cc:374] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:39.407320 22191 raft_consensus.cc:729] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0652df4464d54fe7863ddfbf5a14b10e, State: Initialized, Role: FOLLOWER
I20260810 03:18:39.407611 22191 consensus_queue.cc:260] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.408032 22196 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.408313 22196 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:39.408735 22191 ts_tablet_manager.cc:1428] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 03:18:39.469786 22196 raft_consensus.cc:482] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:39.470134 22196 raft_consensus.cc:504] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.470890 22196 leader_election.cc:288] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649), db2c7817be0c4677822f076311b8c825 (127.19.246.194:40475)
I20260810 03:18:39.471588 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1221c129403949b19420833ece16e685" candidate_uuid: "0652df4464d54fe7863ddfbf5a14b10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2c7817be0c4677822f076311b8c825" is_pre_election: true
I20260810 03:18:39.471668 21820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1221c129403949b19420833ece16e685" candidate_uuid: "0652df4464d54fe7863ddfbf5a14b10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bc7159923b414191572b41ba1c81f2" is_pre_election: true
I20260810 03:18:39.471864 21971 raft_consensus.cc:2454] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0652df4464d54fe7863ddfbf5a14b10e in term 0.
I20260810 03:18:39.472051 21820 raft_consensus.cc:2454] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0652df4464d54fe7863ddfbf5a14b10e in term 0.
I20260810 03:18:39.472606 22042 leader_election.cc:302] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00bc7159923b414191572b41ba1c81f2, 0652df4464d54fe7863ddfbf5a14b10e, db2c7817be0c4677822f076311b8c825; no voters:
I20260810 03:18:39.473033 22196 raft_consensus.cc:2781] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:39.473158 22196 raft_consensus.cc:482] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:39.473279 22196 raft_consensus.cc:3037] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:39.475430 22196 raft_consensus.cc:504] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.476071 22196 leader_election.cc:288] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [CANDIDATE]: Term 1 election: Requested vote from peers 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649), db2c7817be0c4677822f076311b8c825 (127.19.246.194:40475)
I20260810 03:18:39.476614 21971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1221c129403949b19420833ece16e685" candidate_uuid: "0652df4464d54fe7863ddfbf5a14b10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2c7817be0c4677822f076311b8c825"
I20260810 03:18:39.476864 21971 raft_consensus.cc:3037] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:39.476764 21820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1221c129403949b19420833ece16e685" candidate_uuid: "0652df4464d54fe7863ddfbf5a14b10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bc7159923b414191572b41ba1c81f2"
I20260810 03:18:39.477082 21820 raft_consensus.cc:3037] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:39.478950 21971 raft_consensus.cc:2454] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0652df4464d54fe7863ddfbf5a14b10e in term 1.
I20260810 03:18:39.479564 22044 leader_election.cc:302] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0652df4464d54fe7863ddfbf5a14b10e, db2c7817be0c4677822f076311b8c825; no voters:
I20260810 03:18:39.479609 21820 raft_consensus.cc:2454] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0652df4464d54fe7863ddfbf5a14b10e in term 1.
I20260810 03:18:39.480028 22196 raft_consensus.cc:2781] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:39.480280 22196 raft_consensus.cc:686] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [term 1 LEADER]: Becoming Leader. State: Replica: 0652df4464d54fe7863ddfbf5a14b10e, State: Running, Role: LEADER
I20260810 03:18:39.480753 22196 consensus_queue.cc:237] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } }
I20260810 03:18:39.482451 22225 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: TxnStatusTablet state changed. Reason: New leader 0652df4464d54fe7863ddfbf5a14b10e. Latest consensus state: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.482740 22225 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:18:39.483544 22239 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:18:39.483915 22239 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:18:39.484382 21644 catalog_manager.cc:5495] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0652df4464d54fe7863ddfbf5a14b10e (127.19.246.195). New cstate: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } health_report { overall_health: HEALTHY } } }
I20260810 03:18:39.509564 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.501754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:41615 (local address 127.0.0.1:37428)
0810 03:18:39.502034 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:39.502058 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:39.502238 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:39.503128 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:39.503132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:39.503143 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:18:39.503308 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.503316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.505165 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.505169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.506358 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.506372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.507509 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.507514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.507725 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.507794 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:18:39.508978 (+ 1184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:18:39.509003 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:18:39.509087 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 03:18:39.509239 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 03:18:39.509730 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.501901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.19.246.195:41615)
0810 03:18:39.502658 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.502664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.502693 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:18:39.502790 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.502795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.502900 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.503088 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:39.503534 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.504984 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.506520 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.507270 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.507464 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.508000 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:39.508831 (+ 831us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:39.508991 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.509166 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.509271 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":425,"threads_started":1}
I20260810 03:18:39.516166 21820 raft_consensus.cc:1261] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Refusing update from remote peer 0652df4464d54fe7863ddfbf5a14b10e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:39.516431 21971 raft_consensus.cc:1261] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Refusing update from remote peer 0652df4464d54fe7863ddfbf5a14b10e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:39.516955 22225 consensus_queue.cc:1035] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:39.517354 22196 consensus_queue.cc:1035] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:39.525774 22199 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: TxnStatusTablet state changed. Reason: New leader 0652df4464d54fe7863ddfbf5a14b10e. Latest consensus state: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.526242 22199 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:39.529567 22197 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: TxnStatusTablet state changed. Reason: New leader 0652df4464d54fe7863ddfbf5a14b10e. Latest consensus state: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.529913 22197 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:39.530267 22243 mvcc.cc:205] Tried to move back new op lower bound from 7316815542328852480 to 7316815542195687424. Current Snapshot: MvccSnapshot[applied={T|T < 7316815542328852480}]
I20260810 03:18:39.532373 22199 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.532672 22199 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P db2c7817be0c4677822f076311b8c825: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:39.538920 22197 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.539307 22197 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P 00bc7159923b414191572b41ba1c81f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:39.541584 22225 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.541898 22225 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:18:39.542855 22196 tablet_replica.cc:439] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0652df4464d54fe7863ddfbf5a14b10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 } } peers { permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 } } peers { permanent_uuid: "0652df4464d54fe7863ddfbf5a14b10e" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41615 } } }
I20260810 03:18:39.543244 22196 tablet_replica.cc:442] T 1221c129403949b19420833ece16e685 P 0652df4464d54fe7863ddfbf5a14b10e: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:18:39.558578 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.550688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.19.246.195:41615)
0810 03:18:39.550927 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.550937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.551433 (+ 496us) server_negotiation.cc:407] Connection header received
0810 03:18:39.551602 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.551607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.551684 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.551760 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:39.552404 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.553228 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.556295 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.556792 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.556908 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.557548 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:39.557704 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:39.557793 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.558043 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.558132 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:18:39.581168 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.573603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:41615 (local address 127.19.246.195:53433)
0810 03:18:39.574476 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:39.574492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:39.574640 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:39.575103 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:39.575107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:39.575116 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:18:39.575602 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.575612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.577288 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.577292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.578593 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:39.578605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.579805 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.579810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:39.579934 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.580016 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:18:39.580510 (+ 494us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:18:39.580519 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:18:39.580581 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 03:18:39.580696 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":474,"threads_started":1}
I20260810 03:18:39.581219 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.573572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:53433 (local address 127.19.246.195:41615)
0810 03:18:39.573784 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.573792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.574608 (+ 816us) server_negotiation.cc:407] Connection header received
0810 03:18:39.574815 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.574828 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.574951 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.575087 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:39.575772 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.577100 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.578752 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.579593 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.579759 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.580228 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:39.580386 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:39.580519 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.580639 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.580741 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:18:39.593112 21971 raft_consensus.cc:1261] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Refusing update from remote peer 0652df4464d54fe7863ddfbf5a14b10e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:39.593580 21820 raft_consensus.cc:1261] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Refusing update from remote peer 0652df4464d54fe7863ddfbf5a14b10e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:39.593772 22257 consensus_queue.cc:1035] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db2c7817be0c4677822f076311b8c825" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:39.594278 22225 consensus_queue.cc:1035] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bc7159923b414191572b41ba1c81f2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:39.694600 21643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:39.694890 21643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:39.698773 21643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0495e7719164ec4b336b5042ff12905: Sending DeleteTablet for 3 replicas of tablet a93283c33cf04c77840891c9e09d337c
I20260810 03:18:39.700037 21794 tablet_service.cc:1514] Processing DeleteTablet for tablet a93283c33cf04c77840891c9e09d337c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:39 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:37502
I20260810 03:18:39.700053 21945 tablet_service.cc:1514] Processing DeleteTablet for tablet a93283c33cf04c77840891c9e09d337c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:39 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:39520
I20260810 03:18:39.700544 22094 tablet_service.cc:1514] Processing DeleteTablet for tablet a93283c33cf04c77840891c9e09d337c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:39 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:37412
I20260810 03:18:39.716650 22262 tablet_replica.cc:330] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: stopping tablet replica
I20260810 03:18:39.713166 22264 tablet_replica.cc:330] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: stopping tablet replica
I20260810 03:18:39.716969 22262 raft_consensus.cc:2229] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:39.717204 22264 raft_consensus.cc:2229] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:18:39.717337 22262 raft_consensus.cc:2258] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:39.718057 22264 raft_consensus.cc:2258] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:39.718792 22262 ts_tablet_manager.cc:1905] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:39.718907 22263 tablet_replica.cc:330] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: stopping tablet replica
I20260810 03:18:39.719137 22263 raft_consensus.cc:2229] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:39.719460 22263 raft_consensus.cc:2258] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:39.719753 22264 ts_tablet_manager.cc:1905] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:39.720685 22263 ts_tablet_manager.cc:1905] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:39.727689 22262 ts_tablet_manager.cc:1918] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:39.727866 22262 log.cc:1196] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/a93283c33cf04c77840891c9e09d337c
I20260810 03:18:39.728431 22262 ts_tablet_manager.cc:1939] T a93283c33cf04c77840891c9e09d337c P db2c7817be0c4677822f076311b8c825: Deleting consensus metadata
I20260810 03:18:39.728679 22263 ts_tablet_manager.cc:1918] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:39.728960 22263 log.cc:1196] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/a93283c33cf04c77840891c9e09d337c
I20260810 03:18:39.729581 22263 ts_tablet_manager.cc:1939] T a93283c33cf04c77840891c9e09d337c P 00bc7159923b414191572b41ba1c81f2: Deleting consensus metadata
I20260810 03:18:39.729808 21631 catalog_manager.cc:4841] TS db2c7817be0c4677822f076311b8c825 (127.19.246.194:40475): tablet a93283c33cf04c77840891c9e09d337c (table test-table [id=ce5f47bbb6664d1f9511e21946fc8717]) successfully deleted
I20260810 03:18:39.730809 21629 catalog_manager.cc:4841] TS 00bc7159923b414191572b41ba1c81f2 (127.19.246.193:43649): tablet a93283c33cf04c77840891c9e09d337c (table test-table [id=ce5f47bbb6664d1f9511e21946fc8717]) successfully deleted
I20260810 03:18:39.732988 22264 ts_tablet_manager.cc:1918] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:39.733369 22264 log.cc:1196] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/a93283c33cf04c77840891c9e09d337c
I20260810 03:18:39.733887 22264 ts_tablet_manager.cc:1939] T a93283c33cf04c77840891c9e09d337c P 0652df4464d54fe7863ddfbf5a14b10e: Deleting consensus metadata
I20260810 03:18:39.734985 21628 catalog_manager.cc:4841] TS 0652df4464d54fe7863ddfbf5a14b10e (127.19.246.195:41615): tablet a93283c33cf04c77840891c9e09d337c (table test-table [id=ce5f47bbb6664d1f9511e21946fc8717]) successfully deleted
Aug 10 03:18:39 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:18:39.749352 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.715555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.19.246.193:43649)
0810 03:18:39.715842 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.715851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.725589 (+ 9738us) server_negotiation.cc:407] Connection header received
0810 03:18:39.727848 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.727856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.728020 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.728612 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:39.733267 (+ 4655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.734677 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.737062 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.737529 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.737654 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.742891 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:39.742908 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:39.742925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:39.742968 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:39.746276 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.747117 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.747125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.747128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.747275 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.747749 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.747753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.747755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.748120 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:39.748444 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.748749 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.748981 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260810 03:18:39.750178 21867 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37518
I20260810 03:18:39.771198 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.753472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.19.246.254:42381)
0810 03:18:39.753743 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.753748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.753906 (+ 158us) server_negotiation.cc:407] Connection header received
0810 03:18:39.754072 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.754077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.754203 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.754359 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:39.756341 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.757611 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.759530 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.760290 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.760415 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.764115 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:39.764124 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:39.764129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:39.764178 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:39.767985 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.768941 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.768947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.768951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.769105 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.769545 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.769551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.769554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.769929 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:39.770179 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.770590 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.770850 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
W20260810 03:18:39.771993 21643 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57172
I20260810 03:18:39.774369 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21723
I20260810 03:18:39.794179 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.775633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.19.246.193:43649)
0810 03:18:39.776510 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.776517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.781262 (+ 4745us) server_negotiation.cc:407] Connection header received
0810 03:18:39.781496 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.781502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.781595 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.781726 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:39.783309 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.784138 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.785415 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.785847 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.785995 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.788503 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:39.788508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:39.788511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:39.788552 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:39.791457 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.792178 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.792198 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.792201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.792332 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.792739 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.792748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.792750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.793017 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:39.793303 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.793598 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.793830 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726}
W20260810 03:18:39.794759 21867 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37528
W20260810 03:18:39.795429 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:18:39.795612 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 21723
I20260810 03:18:39.823946 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21876
Aug 10 03:18:39 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:18:39.841647 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.824874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.19.246.194:40475)
0810 03:18:39.825407 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.825412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.825428 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:18:39.825481 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.825485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.825560 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.825687 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:39.826991 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.827933 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.829502 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.829979 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.830137 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.835050 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:39.835068 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:39.835082 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:39.835126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:39.838431 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.839330 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.839339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.839341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.839568 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.840001 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.840007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.840017 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.840392 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:39.840724 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.841030 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.841272 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":256,"threads_started":1}
W20260810 03:18:39.842337 22016 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39540
W20260810 03:18:39.843103 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:18:39.843469 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 21876
I20260810 03:18:39.880673 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22025
Aug 10 03:18:39 dist-test-slave-btw8 krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:18:39.901638 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:39.881602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.19.246.195:41615)
0810 03:18:39.881854 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:39.881861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:39.881930 (+ 69us) server_negotiation.cc:407] Connection header received
0810 03:18:39.882013 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:39.882019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:39.882108 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:39.882234 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:39.883870 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.885264 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.886759 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:39.887805 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:39.887968 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:39.893199 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:39.893213 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:39.893231 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:39.893278 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:39.897088 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.898257 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.898265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.898268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.898465 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:39.899051 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:39.899055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:39.899058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:39.899525 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:39.900022 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:39.900636 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 03:18:39.901124 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 03:18:39.902470 22163 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450
W20260810 03:18:39.903494 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22025: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:18:39.903779 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 22025
I20260810 03:18:39.929667 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 21611
I20260810 03:18:39.930903 21688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:40.052784 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 21611
2026-08-10T03:18:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5241 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:18:40 dist-test-slave-btw8 krb5kdc[22276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:18:40 dist-test-slave-btw8 krb5kdc[22276](info): set up 2 sockets
Aug 10 03:18:40 dist-test-slave-btw8 krb5kdc[22276](info): commencing operation
krb5kdc: starting...
W20260810 03:18:41.236924 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:18:41 dist-test-slave-btw8 krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:18:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:18:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:18:41.484750 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:39681
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:39681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:41.835561 22290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:41.835883 22290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:41.835978 22290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:41.843709 22290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:18:41.843892 22290 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:18:41.843936 22290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:41.843991 22290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:41.844034 22290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:18:41.844100 22290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:18:41.854529 22290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:39681
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:39681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:41.855955 22290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:41.857383 22290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:41.869570 22295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:41.869594 22298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:41.869594 22296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:41.870180 22290 server_base.cc:1034] running on GCE node
I20260810 03:18:41.871057 22290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:41.872650 22290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:41.873917 22290 hybrid_clock.cc:648] HybridClock initialized: now 1786331921873862 us; error 34 us; skew 500 ppm
I20260810 03:18:41.874286 22290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:41 dist-test-slave-btw8 krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331921, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:41.877969 22290 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:18:41.879777 22290 webserver.cc:460] Webserver started at http://127.19.246.254:34987/ using document root <none> and password file <none>
I20260810 03:18:41.880584 22290 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:41.880733 22290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:41.881073 22290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:41.884326 22290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "ad11732858494c28abc6f6e5f1f01ed2"
format_stamp: "Formatted at 2026-08-10 03:18:41 on dist-test-slave-btw8"
server_key: "2b643996e223421cf772d52541d7e432"
server_key_iv: "1ad389c7dc452867416e38374a6cc933"
server_key_version: "encryptionkey@0"
I20260810 03:18:41.885161 22290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "ad11732858494c28abc6f6e5f1f01ed2"
format_stamp: "Formatted at 2026-08-10 03:18:41 on dist-test-slave-btw8"
server_key: "2b643996e223421cf772d52541d7e432"
server_key_iv: "1ad389c7dc452867416e38374a6cc933"
server_key_version: "encryptionkey@0"
I20260810 03:18:41.890828 22290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:18:41.894631 22305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:41.896162 22290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:18:41.896366 22290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "ad11732858494c28abc6f6e5f1f01ed2"
format_stamp: "Formatted at 2026-08-10 03:18:41 on dist-test-slave-btw8"
server_key: "2b643996e223421cf772d52541d7e432"
server_key_iv: "1ad389c7dc452867416e38374a6cc933"
server_key_version: "encryptionkey@0"
I20260810 03:18:41.896610 22290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:41.914551 22290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:41.919723 22290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:41.920138 22290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:41.944466 22290 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:39681
I20260810 03:18:41.944484 22374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:39681 every 8 connection(s)
I20260810 03:18:41.945989 22290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:18:41.949568 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 22290
I20260810 03:18:41.949808 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:18:41.950304 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:41.950687 20443 external_mini_cluster.cc:1428] Setting key 014e13bcc8096836dd58ff0f6bfdce18
I20260810 03:18:41.967439 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: Bootstrap starting.
Aug 10 03:18:41 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:41.971148 22375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:41.972385 22375 log.cc:824] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:41.975517 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: No bootstrap required, opened a new log
I20260810 03:18:41.977492 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:41.952957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.19.246.254:39681)
0810 03:18:41.953808 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:41.953824 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:41.953865 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:18:41.955987 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:41.956019 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:41.956485 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:41.956908 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:41.959053 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:41.960887 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:41.962983 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:41.963970 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:41.964145 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:41.969389 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:41.969400 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:41.969416 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:41.969462 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:41.972993 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:41.973850 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:41.973859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:41.973866 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:41.974048 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:41.974503 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:41.974510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:41.974517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:41.975083 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:41.975446 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:41.975928 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 03:18:41.976413 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":397,"threads_started":1}
I20260810 03:18:41.983858 22375 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad11732858494c28abc6f6e5f1f01ed2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39681 } }
I20260810 03:18:41.984342 22375 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:41.984452 22375 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad11732858494c28abc6f6e5f1f01ed2, State: Initialized, Role: FOLLOWER
I20260810 03:18:41.985083 22375 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad11732858494c28abc6f6e5f1f01ed2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39681 } }
I20260810 03:18:41.985386 22375 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:18:41.985538 22375 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:18:41.985819 22375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:41.988395 22375 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad11732858494c28abc6f6e5f1f01ed2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39681 } }
I20260810 03:18:41.988971 22375 leader_election.cc:302] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad11732858494c28abc6f6e5f1f01ed2; no voters:
I20260810 03:18:41.989569 22375 leader_election.cc:288] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:18:41.989770 22380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:41.990547 22380 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [term 1 LEADER]: Becoming Leader. State: Replica: ad11732858494c28abc6f6e5f1f01ed2, State: Running, Role: LEADER
I20260810 03:18:41.991158 22380 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad11732858494c28abc6f6e5f1f01ed2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39681 } }
I20260810 03:18:41.992234 22375 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:18:41.994330 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad11732858494c28abc6f6e5f1f01ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad11732858494c28abc6f6e5f1f01ed2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39681 } } }
I20260810 03:18:41.994814 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:41.995282 22382 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad11732858494c28abc6f6e5f1f01ed2. Latest consensus state: current_term: 1 leader_uuid: "ad11732858494c28abc6f6e5f1f01ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad11732858494c28abc6f6e5f1f01ed2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39681 } } }
I20260810 03:18:41.995587 22382 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:42.000990 22388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:18:42.006560 22388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:18:42.017174 22388 catalog_manager.cc:1347] Generated new cluster ID: 670c9448e7ef4d9aad20137be86b70d7
I20260810 03:18:42.017326 22388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:18:42.057947 22388 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:18:42.059656 22388 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:18:42.066985 22388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: Generated new TSK 0
I20260810 03:18:42.068087 22388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:18:42.131433 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39681
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:42.483000 22401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:42.483304 22401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:42.483517 22401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:42.491216 22401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:42.491443 22401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:42.491634 22401 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:18:42.491729 22401 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:18:42.491842 22401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:18:42.502840 22401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:39681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:42.504329 22401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:42.505697 22401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:42.519639 22409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:42.519629 22407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:42.519663 22406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:42.520717 22401 server_base.cc:1034] running on GCE node
I20260810 03:18:42.521459 22401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:42.522490 22401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:42.523739 22401 hybrid_clock.cc:648] HybridClock initialized: now 1786331922523691 us; error 43 us; skew 500 ppm
I20260810 03:18:42.524107 22401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:42 dist-test-slave-btw8 krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:42.532804 22401 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:18:42.534590 22401 webserver.cc:460] Webserver started at http://127.19.246.193:33231/ using document root <none> and password file <none>
I20260810 03:18:42.535315 22401 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:42.535511 22401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:42.535892 22401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:42.538976 22401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "ef888aa00e35409d964caf74d09d5eff"
format_stamp: "Formatted at 2026-08-10 03:18:42 on dist-test-slave-btw8"
server_key: "015664bff2efeb05458cebe466f9a5d6"
server_key_iv: "c1688b90c8b953745527c5244759de79"
server_key_version: "encryptionkey@0"
I20260810 03:18:42.539822 22401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "ef888aa00e35409d964caf74d09d5eff"
format_stamp: "Formatted at 2026-08-10 03:18:42 on dist-test-slave-btw8"
server_key: "015664bff2efeb05458cebe466f9a5d6"
server_key_iv: "c1688b90c8b953745527c5244759de79"
server_key_version: "encryptionkey@0"
I20260810 03:18:42.545504 22401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:18:42.549396 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:42.550904 22401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:18:42.551133 22401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "ef888aa00e35409d964caf74d09d5eff"
format_stamp: "Formatted at 2026-08-10 03:18:42 on dist-test-slave-btw8"
server_key: "015664bff2efeb05458cebe466f9a5d6"
server_key_iv: "c1688b90c8b953745527c5244759de79"
server_key_version: "encryptionkey@0"
I20260810 03:18:42.551415 22401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:42.579603 22401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:42.584285 22401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:42.584666 22401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:42.587697 22401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:42.587832 22401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:42.587978 22401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:42.588094 22401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:18:42 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:42.618674 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:42.588228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:34611 (local address 127.19.246.254:39681)
0810 03:18:42.591319 (+ 3091us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:42.591324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:42.591344 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:18:42.594192 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:42.594203 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:42.594320 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:42.594438 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:42.597653 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:42.598409 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:42.602132 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:42.602794 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:42.602959 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:42.609341 (+ 6382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:42.609348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:42.609354 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:42.609412 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:42.613843 (+ 4431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:42.615463 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:42.615472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:42.615477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:42.615707 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:42.616462 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:42.616467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:42.616471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:42.616888 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:42.617062 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:42.618034 (+ 972us) server_negotiation.cc:299] Negotiation successful
0810 03:18:42.618302 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2968,"thread_start_us":2848,"threads_started":1}
I20260810 03:18:42.619943 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:42.589111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39681 (local address 127.19.246.193:34611)
0810 03:18:42.590287 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:42.590345 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:42.593725 (+ 3380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:42.594834 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:42.594853 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:42.596083 (+ 1230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:42.597383 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:42.597425 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:42.598605 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:42.598614 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:42.601849 (+ 3235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:42.601874 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:42.603334 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:42.603340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:42.603620 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:42.604665 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:42.604736 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:42.608889 (+ 4153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:42.614141 (+ 5252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:42.614160 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:42.614190 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:42.615219 (+ 1029us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:42.615851 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:42.615856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:42.615860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:42.616327 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:42.617146 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:42.617154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:42.617576 (+ 422us) client_negotiation.cc:769] Sending connection context
0810 03:18:42.617871 (+ 295us) client_negotiation.cc:240] Negotiation successful
0810 03:18:42.618460 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":482,"threads_started":1}
I20260810 03:18:42.643065 22401 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:45521
I20260810 03:18:42.643175 22547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:45521 every 8 connection(s)
I20260810 03:18:42.645102 22401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:18:42.646230 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 22401
I20260810 03:18:42.646497 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:18:42.647360 20443 external_mini_cluster.cc:1428] Setting key 2b7c4e95d8c5c12f6fa6c1ce4cd38ffc
I20260810 03:18:42.647658 22548 heartbeater.cc:347] Connected to a master server at 127.19.246.254:39681
I20260810 03:18:42.647905 22548 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:42.648813 22548 heartbeater.cc:511] Master 127.19.246.254:39681 requested a full tablet report, sending...
I20260810 03:18:42.651386 22324 ts_manager.cc:194] Registered new tserver with Master: ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521)
I20260810 03:18:42.651712 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:42.628268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:33719 (local address 127.19.246.254:39681)
0810 03:18:42.628658 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:42.628670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:42.632303 (+ 3633us) server_negotiation.cc:407] Connection header received
0810 03:18:42.632520 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:42.632524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:42.632596 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:42.632688 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:42.634535 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:42.635859 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:42.639093 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:42.639920 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:42.640068 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:42.642716 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:42.642726 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:42.642731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:42.642777 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:42.646431 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:42.648972 (+ 2541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:42.648977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:42.648980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:42.649150 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:42.650070 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:42.650075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:42.650079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:42.650385 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:42.650587 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:42.650997 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 03:18:42.651258 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 03:18:42.651763 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:42.627937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39681 (local address 127.19.246.193:33719)
0810 03:18:42.632128 (+ 4191us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:42.632162 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:42.632367 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:42.632759 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:42.632766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:42.633600 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:42.634298 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:42.634316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:42.636040 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:42.636049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:42.638545 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:42.638576 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:42.640197 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:42.640205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:42.640371 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:42.641090 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:42.641114 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:42.642446 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:42.648190 (+ 5744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:42.648198 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:42.648208 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:42.648824 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:42.649748 (+ 924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:42.649751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:42.649754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:42.649934 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:42.650526 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:42.650531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:42.650730 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 03:18:42.650880 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 03:18:42.651255 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4012,"thread_start_us":3775,"threads_started":1}
I20260810 03:18:42.653887 22324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:34611
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:18:42.703263 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39681
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:43.054382 22552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:43.054735 22552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:43.054901 22552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:43.062757 22552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:43.062947 22552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:43.063117 22552 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:18:43.063244 22552 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:18:43.063421 22552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:18:43.074020 22552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:39681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:43.075589 22552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:43.076982 22552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:43.090761 22560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:43.090796 22558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:43.090720 22557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:43.092109 22552 server_base.cc:1034] running on GCE node
I20260810 03:18:43.092839 22552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:43.093940 22552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:43.095216 22552 hybrid_clock.cc:648] HybridClock initialized: now 1786331923095157 us; error 66 us; skew 500 ppm
I20260810 03:18:43.095645 22552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:43 dist-test-slave-btw8 krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331923, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:43.099213 22552 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:18:43.100940 22552 webserver.cc:460] Webserver started at http://127.19.246.194:46607/ using document root <none> and password file <none>
I20260810 03:18:43.101754 22552 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:43.101910 22552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:43.102249 22552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:43.105504 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "b0b66e0fa6d845dab4da4a135f54115b"
format_stamp: "Formatted at 2026-08-10 03:18:43 on dist-test-slave-btw8"
server_key: "c24e2868b66798c732af46c44c49b17d"
server_key_iv: "5c2999ec0058ed5423063f3cc46a4e3f"
server_key_version: "encryptionkey@0"
I20260810 03:18:43.106365 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "b0b66e0fa6d845dab4da4a135f54115b"
format_stamp: "Formatted at 2026-08-10 03:18:43 on dist-test-slave-btw8"
server_key: "c24e2868b66798c732af46c44c49b17d"
server_key_iv: "5c2999ec0058ed5423063f3cc46a4e3f"
server_key_version: "encryptionkey@0"
I20260810 03:18:43.112632 22552 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 03:18:43.116482 22568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:43.118088 22552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:18:43.118300 22552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "b0b66e0fa6d845dab4da4a135f54115b"
format_stamp: "Formatted at 2026-08-10 03:18:43 on dist-test-slave-btw8"
server_key: "c24e2868b66798c732af46c44c49b17d"
server_key_iv: "5c2999ec0058ed5423063f3cc46a4e3f"
server_key_version: "encryptionkey@0"
I20260810 03:18:43.118543 22552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:43.136387 22552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:43.141706 22552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:43.142056 22552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:43.145171 22552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:43.145308 22552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:43.145442 22552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:43.145557 22552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:18:43 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331923, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:43.177775 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.145717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:40471 (local address 127.19.246.254:39681)
0810 03:18:43.145935 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:43.145945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:43.147708 (+ 1763us) server_negotiation.cc:407] Connection header received
0810 03:18:43.150333 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:43.150338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:43.150446 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:43.150576 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:43.153115 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.154742 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.158716 (+ 3974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.159541 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.159722 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.167309 (+ 7587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:43.167320 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:43.167329 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:43.167427 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:43.171574 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.173730 (+ 2156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.173739 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.173745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.173936 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.174758 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.174765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.174770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.175237 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:43.175445 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:43.176957 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0810 03:18:43.177293 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:18:43.182984 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.146419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39681 (local address 127.19.246.194:40471)
0810 03:18:43.147486 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:43.147541 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:43.150026 (+ 2485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:43.150879 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:43.150893 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:43.151836 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:43.152789 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.152826 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.155007 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.155012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.158472 (+ 3460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.158499 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.159967 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.159999 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.160472 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.161599 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:43.161652 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:43.166859 (+ 5207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:43.171874 (+ 5015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:43.171886 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:43.171908 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:43.173412 (+ 1504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:43.174144 (+ 732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:43.174153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:43.174159 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:43.174575 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:43.175580 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:43.175596 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:43.176304 (+ 708us) client_negotiation.cc:769] Sending connection context
0810 03:18:43.178472 (+ 2168us) client_negotiation.cc:240] Negotiation successful
0810 03:18:43.180124 (+ 1652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":429,"threads_started":1}
I20260810 03:18:43.195209 22552 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:41735
I20260810 03:18:43.195335 22694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:41735 every 8 connection(s)
I20260810 03:18:43.196698 22552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:18:43.199566 22695 heartbeater.cc:347] Connected to a master server at 127.19.246.254:39681
I20260810 03:18:43.199910 22695 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:43.200450 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 22552
I20260810 03:18:43.200670 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:18:43.201428 20443 external_mini_cluster.cc:1428] Setting key e86402429c4db2ed18856cee66639b57
I20260810 03:18:43.205579 22695 heartbeater.cc:511] Master 127.19.246.254:39681 requested a full tablet report, sending...
I20260810 03:18:43.207708 22324 ts_manager.cc:194] Registered new tserver with Master: b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735)
I20260810 03:18:43.209081 22324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:40471
I20260810 03:18:43.214428 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.192985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:48803 (local address 127.19.246.254:39681)
0810 03:18:43.193208 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:43.193213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:43.194476 (+ 1263us) server_negotiation.cc:407] Connection header received
0810 03:18:43.194554 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:43.194557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:43.194639 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:43.194766 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:43.196075 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.196849 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.198745 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.199440 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.199576 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.203078 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:43.203090 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:43.203099 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:43.203180 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:43.209677 (+ 6497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.210927 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.210938 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.210941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.211200 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.211916 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.211925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.211930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.212619 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:43.212978 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:43.213511 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 03:18:43.214081 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 03:18:43.214461 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.192966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39681 (local address 127.19.246.194:48803)
0810 03:18:43.194077 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:43.194104 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:43.194328 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:43.194797 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:43.194801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:43.195539 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:43.195893 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.195903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.196994 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.197011 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.198581 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.198595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.199725 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.199734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.199958 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.200857 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:43.200888 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:43.202276 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:43.209903 (+ 7627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:43.209913 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:43.209917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:43.210753 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:43.211417 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:43.211426 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:43.211434 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:43.211767 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:43.212908 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:43.212913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:43.213172 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 03:18:43.213367 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 03:18:43.213779 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":444,"threads_started":1}
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:18:43.265548 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39681
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:43.629365 22699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:43.629727 22699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:43.629910 22699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:43.639827 22699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:43.640017 22699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:43.640192 22699 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:18:43.640291 22699 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:18:43.640403 22699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:18:43.651949 22699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:39681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:43.653430 22699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:43.654887 22699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:18:43.657414 22548 heartbeater.cc:503] Master 127.19.246.254:39681 was elected leader, sending a full tablet report...
W20260810 03:18:43.670717 22705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:43.670722 22706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:43.670671 22708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:43.672089 22699 server_base.cc:1034] running on GCE node
I20260810 03:18:43.672806 22699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:43.673900 22699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:43.675148 22699 hybrid_clock.cc:648] HybridClock initialized: now 1786331923675072 us; error 63 us; skew 500 ppm
I20260810 03:18:43.675580 22699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:43 dist-test-slave-btw8 krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331923, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:43.679288 22699 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:18:43.681020 22699 webserver.cc:460] Webserver started at http://127.19.246.195:44111/ using document root <none> and password file <none>
I20260810 03:18:43.681824 22699 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:43.681977 22699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:43.682365 22699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:43.685930 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "0e68e0fd34df4ab1bca59d36228cb248"
format_stamp: "Formatted at 2026-08-10 03:18:43 on dist-test-slave-btw8"
server_key: "14b87f4ff558936ff1b7fb9c3a1e6489"
server_key_iv: "926f0d441cdcc2d20f5334823f8cfa41"
server_key_version: "encryptionkey@0"
I20260810 03:18:43.686939 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "0e68e0fd34df4ab1bca59d36228cb248"
format_stamp: "Formatted at 2026-08-10 03:18:43 on dist-test-slave-btw8"
server_key: "14b87f4ff558936ff1b7fb9c3a1e6489"
server_key_iv: "926f0d441cdcc2d20f5334823f8cfa41"
server_key_version: "encryptionkey@0"
I20260810 03:18:43.693383 22699 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260810 03:18:43.697711 22715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:43.699841 22699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:18:43.700079 22699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "0e68e0fd34df4ab1bca59d36228cb248"
format_stamp: "Formatted at 2026-08-10 03:18:43 on dist-test-slave-btw8"
server_key: "14b87f4ff558936ff1b7fb9c3a1e6489"
server_key_iv: "926f0d441cdcc2d20f5334823f8cfa41"
server_key_version: "encryptionkey@0"
I20260810 03:18:43.700366 22699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:43.726243 22699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:43.731033 22699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:43.731506 22699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:43.734475 22699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:43.734649 22699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:43.734799 22699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:43.734898 22699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:18:43 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331923, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:43.767231 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.735273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:60117 (local address 127.19.246.254:39681)
0810 03:18:43.735913 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:43.735920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:43.737199 (+ 1279us) server_negotiation.cc:407] Connection header received
0810 03:18:43.739421 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:43.739430 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:43.739545 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:43.739686 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:43.743187 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.744696 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.748031 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.748889 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.749092 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.756132 (+ 7040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:43.756140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:43.756147 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:43.756201 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:43.760963 (+ 4762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.762633 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.762638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.762641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.762807 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.763616 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.763623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.763629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.764181 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:43.764398 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:43.766091 (+ 1693us) server_negotiation.cc:299] Negotiation successful
0810 03:18:43.766680 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":386,"threads_started":1}
I20260810 03:18:43.769073 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.736003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39681 (local address 127.19.246.195:60117)
0810 03:18:43.736982 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:43.737037 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:43.739054 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:43.740281 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:43.740298 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:43.741569 (+ 1271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:43.742857 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.742896 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.744912 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.744924 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.747763 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.747805 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.749225 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.749234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.749679 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.750878 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:43.750925 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:43.755768 (+ 4843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:43.761251 (+ 5483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:43.761269 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:43.761297 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:43.762377 (+ 1080us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:43.762979 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:43.762987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:43.763007 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:43.763413 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:43.764401 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:43.764415 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:43.765348 (+ 933us) client_negotiation.cc:769] Sending connection context
0810 03:18:43.765887 (+ 539us) client_negotiation.cc:240] Negotiation successful
0810 03:18:43.766756 (+ 869us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":418,"threads_started":1}
I20260810 03:18:43.782476 22699 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:36001
I20260810 03:18:43.782591 22843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:36001 every 8 connection(s)
I20260810 03:18:43.784171 22699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:18:43.786980 22845 heartbeater.cc:347] Connected to a master server at 127.19.246.254:39681
I20260810 03:18:43.787498 22845 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:43.788426 22845 heartbeater.cc:511] Master 127.19.246.254:39681 requested a full tablet report, sending...
I20260810 03:18:43.790249 22324 ts_manager.cc:194] Registered new tserver with Master: 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001)
I20260810 03:18:43.791118 22324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:60117
I20260810 03:18:43.792393 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 22699
I20260810 03:18:43.792603 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:18:43.793331 20443 external_mini_cluster.cc:1428] Setting key 3e925565df72b945db9dd1b610344ea3
I20260810 03:18:43.797075 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:18:43.801272 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.784131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:58851 (local address 127.19.246.254:39681)
0810 03:18:43.784408 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:43.784416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:43.787288 (+ 2872us) server_negotiation.cc:407] Connection header received
0810 03:18:43.787570 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:43.787578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:43.787693 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:43.787799 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:43.788962 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.789919 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.791788 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.792523 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.792658 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.795235 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:43.795241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:43.795244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:43.795277 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:43.798274 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.798999 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.799004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.799008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.799158 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.799621 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.799627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.799630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.799937 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:43.800140 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:43.800623 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 03:18:43.800881 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 03:18:43.801355 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.784167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39681 (local address 127.19.246.195:58851)
0810 03:18:43.787176 (+ 3009us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:43.787207 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:43.787405 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:43.787874 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:43.787883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:43.788436 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:43.788804 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.788814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.790083 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.790088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.791601 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.791630 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.792795 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.792805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.793045 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.793807 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:43.793829 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:43.795013 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:43.798418 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:43.798428 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:43.798432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:43.798863 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:43.799304 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:43.799308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:43.799312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:43.799510 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:43.800113 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:43.800118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:43.800352 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 03:18:43.800516 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 03:18:43.800881 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2803,"mutex_wait_us":1,"thread_start_us":2643,"threads_started":1}
Aug 10 03:18:43 dist-test-slave-btw8 krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:18:43 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:43.858474 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.842117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.19.246.254:39681)
0810 03:18:43.842326 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:43.842331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:43.842899 (+ 568us) server_negotiation.cc:407] Connection header received
0810 03:18:43.843126 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:43.843131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:43.843216 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:43.843314 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:43.845176 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.846072 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.847517 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.848018 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.848126 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.852622 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:43.852627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:43.852632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:43.852680 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:43.855775 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.856591 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.856596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.856598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.856753 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:43.857170 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:43.857174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:43.857177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:43.857498 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:43.857733 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:43.858007 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 03:18:43.858220 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:18:43.863202 22324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:18:43.870746 22324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:43.898849 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.886214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.19.246.194:41735)
0810 03:18:43.887776 (+ 1562us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:43.887792 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:43.888828 (+ 1036us) server_negotiation.cc:407] Connection header received
0810 03:18:43.888950 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:43.888959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:43.889232 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:43.889426 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:43.890512 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.892141 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.895420 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.896976 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.897352 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.897571 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:43.898281 (+ 710us) server_negotiation.cc:299] Negotiation successful
0810 03:18:43.898499 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1353,"thread_start_us":1217,"threads_started":1}
I20260810 03:18:43.898928 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.885946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:41735 (local address 127.0.0.1:38214)
0810 03:18:43.888345 (+ 2399us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:43.888423 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:43.888629 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:43.889705 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:43.889713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:43.889746 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:43.890296 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.890311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.892307 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.892312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.895162 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.895190 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.897737 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.897744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.897980 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.898043 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 03:18:43.898202 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 03:18:43.898428 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2092,"mutex_wait_us":1,"spinlock_wait_cycles":3072,"thread_start_us":5350,"threads_started":1}
I20260810 03:18:43.900414 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.886858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:45521 (local address 127.0.0.1:38868)
0810 03:18:43.891633 (+ 4775us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:43.891650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:43.891816 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:43.892735 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:43.892743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:43.892769 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:43.893190 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.893203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.895272 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.895278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.897254 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.897269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.899342 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.899404 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.899623 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.899664 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:18:43.899788 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 03:18:43.899981 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4631,"mutex_wait_us":73,"thread_start_us":1636,"threads_started":1}
I20260810 03:18:43.900842 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.888355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.19.246.193:45521)
0810 03:18:43.890553 (+ 2198us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:43.890574 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:43.891772 (+ 1198us) server_negotiation.cc:407] Connection header received
0810 03:18:43.892166 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:43.892183 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:43.892445 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:43.892716 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:43.893392 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.895079 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.897441 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.898745 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.898954 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.899172 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:43.899908 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 03:18:43.900247 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":584,"threads_started":1}
I20260810 03:18:43.904729 22623 tablet_service.cc:1467] Processing CreateTablet for tablet 972b61b4bd874401b4df89339050783b (DEFAULT_TABLE table=test-table [id=bdfdef0edf6b4c29ba52df3d46e19d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:43.906117 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.886684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36001 (local address 127.0.0.1:37326)
0810 03:18:43.888345 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:43.888423 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:43.888629 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:43.891908 (+ 3279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:43.891915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:43.891946 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:43.892606 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.892625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.895187 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.895193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.899505 (+ 4312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:43.899522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.904892 (+ 5370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.904901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:43.905109 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.905147 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:18:43.905371 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 03:18:43.905662 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431,"mutex_wait_us":252,"spinlock_wait_cycles":4992,"thread_start_us":1301,"threads_started":1}
I20260810 03:18:43.906718 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972b61b4bd874401b4df89339050783b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:43.907522 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:43.887816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.19.246.195:36001)
0810 03:18:43.891158 (+ 3342us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:43.891167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:43.891200 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:18:43.891383 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:43.891392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:43.891646 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:43.891895 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:43.892907 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.895017 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.899705 (+ 4688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:43.904227 (+ 4522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:43.904475 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:43.904708 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:43.906527 (+ 1819us) server_negotiation.cc:299] Negotiation successful
0810 03:18:43.906748 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3159,"thread_start_us":2952,"threads_started":1}
I20260810 03:18:43.907671 22473 tablet_service.cc:1467] Processing CreateTablet for tablet 972b61b4bd874401b4df89339050783b (DEFAULT_TABLE table=test-table [id=bdfdef0edf6b4c29ba52df3d46e19d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:43.909286 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972b61b4bd874401b4df89339050783b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:43.912127 22770 tablet_service.cc:1467] Processing CreateTablet for tablet 972b61b4bd874401b4df89339050783b (DEFAULT_TABLE table=test-table [id=bdfdef0edf6b4c29ba52df3d46e19d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:43.913843 22770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972b61b4bd874401b4df89339050783b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:43.923476 22865 tablet_bootstrap.cc:492] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Bootstrap starting.
I20260810 03:18:43.928344 22865 tablet_bootstrap.cc:654] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:43.929769 22865 log.cc:824] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:43.930370 22866 tablet_bootstrap.cc:492] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Bootstrap starting.
I20260810 03:18:43.933565 22868 tablet_bootstrap.cc:492] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Bootstrap starting.
I20260810 03:18:43.934144 22865 tablet_bootstrap.cc:492] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: No bootstrap required, opened a new log
I20260810 03:18:43.934564 22865 ts_tablet_manager.cc:1397] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 03:18:43.935019 22866 tablet_bootstrap.cc:654] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:43.936355 22866 log.cc:824] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:43.937868 22868 tablet_bootstrap.cc:654] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:43.939442 22868 log.cc:824] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:43.940436 22866 tablet_bootstrap.cc:492] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: No bootstrap required, opened a new log
I20260810 03:18:43.940822 22866 ts_tablet_manager.cc:1397] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260810 03:18:43.943477 22868 tablet_bootstrap.cc:492] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: No bootstrap required, opened a new log
I20260810 03:18:43.943825 22868 ts_tablet_manager.cc:1397] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 03:18:43.946867 22865 raft_consensus.cc:348] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
I20260810 03:18:43.947474 22865 raft_consensus.cc:374] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:43.947605 22865 raft_consensus.cc:729] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef888aa00e35409d964caf74d09d5eff, State: Initialized, Role: FOLLOWER
I20260810 03:18:43.948378 22865 consensus_queue.cc:260] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
I20260810 03:18:43.952718 22866 raft_consensus.cc:348] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
I20260810 03:18:43.953291 22866 raft_consensus.cc:374] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:43.953418 22866 raft_consensus.cc:729] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b66e0fa6d845dab4da4a135f54115b, State: Initialized, Role: FOLLOWER
I20260810 03:18:43.954208 22866 consensus_queue.cc:260] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
I20260810 03:18:43.961318 22865 ts_tablet_manager.cc:1428] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Time spent starting tablet: real 0.027s user 0.017s sys 0.008s
I20260810 03:18:43.962460 22868 raft_consensus.cc:348] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
I20260810 03:18:43.963085 22868 raft_consensus.cc:374] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:43.963217 22868 raft_consensus.cc:729] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e68e0fd34df4ab1bca59d36228cb248, State: Initialized, Role: FOLLOWER
I20260810 03:18:43.964150 22868 consensus_queue.cc:260] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
I20260810 03:18:43.971084 22695 heartbeater.cc:503] Master 127.19.246.254:39681 was elected leader, sending a full tablet report...
I20260810 03:18:43.971980 22866 ts_tablet_manager.cc:1428] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Time spent starting tablet: real 0.031s user 0.014s sys 0.018s
I20260810 03:18:43.972652 22845 heartbeater.cc:503] Master 127.19.246.254:39681 was elected leader, sending a full tablet report...
I20260810 03:18:43.973248 22868 ts_tablet_manager.cc:1428] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Time spent starting tablet: real 0.029s user 0.022s sys 0.004s
W20260810 03:18:44.038525 22846 tablet.cc:2369] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:18:44.122943 22873 raft_consensus.cc:482] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:44.123323 22873 raft_consensus.cc:504] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
I20260810 03:18:44.125255 22873 leader_election.cc:288] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001), ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521)
I20260810 03:18:44.133594 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.125777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:39935 (local address 127.19.246.193:45521)
0810 03:18:44.126088 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.126094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.126739 (+ 645us) server_negotiation.cc:407] Connection header received
0810 03:18:44.127016 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.127020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.127088 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.127216 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:44.128336 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.129315 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.131606 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.132410 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.132542 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.132610 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.133142 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.133294 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260810 03:18:44.133819 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.125959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36001 (local address 127.19.246.194:40211)
0810 03:18:44.126983 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.127015 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.127145 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.127646 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.127653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.127677 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:44.128231 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.128247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.129841 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.129849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.131452 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.131474 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.132846 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.132850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.132976 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.133000 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.133123 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.133344 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":113,"thread_start_us":707,"threads_started":1}
I20260810 03:18:44.133888 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.125616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:45521 (local address 127.19.246.194:39935)
0810 03:18:44.126613 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.126631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.126836 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.127262 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.127269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.127306 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:44.128064 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.128081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.129473 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.129479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.131419 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.131443 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.132679 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.132688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.132931 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.132987 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.133116 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.133339 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":96,"thread_start_us":430,"threads_started":1}
I20260810 03:18:44.133613 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.125645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:40211 (local address 127.19.246.195:36001)
0810 03:18:44.125887 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.125894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.127150 (+ 1256us) server_negotiation.cc:407] Connection header received
0810 03:18:44.127306 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.127312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.127434 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.127533 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:44.128462 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.129678 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.131656 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.132614 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.132798 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.132895 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.133148 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.133302 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 03:18:44.134821 22494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972b61b4bd874401b4df89339050783b" candidate_uuid: "b0b66e0fa6d845dab4da4a135f54115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef888aa00e35409d964caf74d09d5eff" is_pre_election: true
I20260810 03:18:44.135035 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972b61b4bd874401b4df89339050783b" candidate_uuid: "b0b66e0fa6d845dab4da4a135f54115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e68e0fd34df4ab1bca59d36228cb248" is_pre_election: true
I20260810 03:18:44.135429 22494 raft_consensus.cc:2454] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0b66e0fa6d845dab4da4a135f54115b in term 0.
I20260810 03:18:44.135656 22792 raft_consensus.cc:2454] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0b66e0fa6d845dab4da4a135f54115b in term 0.
I20260810 03:18:44.136112 22570 leader_election.cc:302] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0b66e0fa6d845dab4da4a135f54115b, ef888aa00e35409d964caf74d09d5eff; no voters:
I20260810 03:18:44.136519 22873 raft_consensus.cc:2781] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:44.136701 22873 raft_consensus.cc:482] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:44.136837 22873 raft_consensus.cc:3037] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.139833 22873 raft_consensus.cc:504] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
I20260810 03:18:44.140632 22873 leader_election.cc:288] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [CANDIDATE]: Term 1 election: Requested vote from peers 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001), ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521)
I20260810 03:18:44.141335 22494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972b61b4bd874401b4df89339050783b" candidate_uuid: "b0b66e0fa6d845dab4da4a135f54115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef888aa00e35409d964caf74d09d5eff"
I20260810 03:18:44.141335 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972b61b4bd874401b4df89339050783b" candidate_uuid: "b0b66e0fa6d845dab4da4a135f54115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e68e0fd34df4ab1bca59d36228cb248"
I20260810 03:18:44.141599 22792 raft_consensus.cc:3037] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.141598 22494 raft_consensus.cc:3037] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.145653 22494 raft_consensus.cc:2454] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0b66e0fa6d845dab4da4a135f54115b in term 1.
I20260810 03:18:44.145653 22792 raft_consensus.cc:2454] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0b66e0fa6d845dab4da4a135f54115b in term 1.
I20260810 03:18:44.146274 22570 leader_election.cc:302] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0b66e0fa6d845dab4da4a135f54115b, ef888aa00e35409d964caf74d09d5eff; no voters:
I20260810 03:18:44.146727 22873 raft_consensus.cc:2781] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:44.147266 22873 raft_consensus.cc:686] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 1 LEADER]: Becoming Leader. State: Replica: b0b66e0fa6d845dab4da4a135f54115b, State: Running, Role: LEADER
I20260810 03:18:44.148032 22873 consensus_queue.cc:237] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } }
W20260810 03:18:44.149083 22549 tablet.cc:2369] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:18:44.154011 22323 catalog_manager.cc:5495] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b reported cstate change: term changed from 0 to 1, leader changed from <none> to b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194). New cstate: current_term: 1 leader_uuid: "b0b66e0fa6d845dab4da4a135f54115b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } health_report { overall_health: UNKNOWN } } }
W20260810 03:18:44.199671 22696 tablet.cc:2369] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 03:18:44 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331921, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:44.229946 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.214567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39681 (local address 127.0.0.1:57084)
0810 03:18:44.215204 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.215220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.215451 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.215834 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.215838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.216820 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:44.217104 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.217114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.218216 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.218223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.219188 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.219200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.220057 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.220063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.220212 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.220953 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:44.220975 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:44.223690 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:44.227113 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:44.227126 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:44.227133 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:44.227791 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:44.228201 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:44.228205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:44.228207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:44.228359 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:44.228917 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:44.228924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:44.229122 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.229266 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.229493 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":355,"threads_started":1}
I20260810 03:18:44.230063 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.214431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.19.246.254:39681)
0810 03:18:44.214678 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.214683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.215320 (+ 637us) server_negotiation.cc:407] Connection header received
0810 03:18:44.215629 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.215634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.215705 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.215813 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:44.217284 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.218072 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.219334 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.219853 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.220001 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.223914 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:44.223920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:44.223934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:44.223968 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:44.226952 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:44.227919 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:44.227924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:44.227926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:44.228070 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:44.228489 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:44.228493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:44.228495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:44.228785 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:44.228896 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.229441 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.229682 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260810 03:18:44.236358 22323 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:57084:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:18:44.237635 22323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:44.246258 22473 tablet_service.cc:1467] Processing CreateTablet for tablet a349b8b4c72840459794644b2919dbc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5477a6b86e342819b5090b7b4819923]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:18:44.246369 22623 tablet_service.cc:1467] Processing CreateTablet for tablet a349b8b4c72840459794644b2919dbc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5477a6b86e342819b5090b7b4819923]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:18:44.246578 22770 tablet_service.cc:1467] Processing CreateTablet for tablet a349b8b4c72840459794644b2919dbc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5477a6b86e342819b5090b7b4819923]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:18:44.247265 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a349b8b4c72840459794644b2919dbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:44.247432 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a349b8b4c72840459794644b2919dbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:44.247450 22770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a349b8b4c72840459794644b2919dbc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:44.252820 22865 tablet_bootstrap.cc:492] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: Bootstrap starting.
I20260810 03:18:44.254420 22865 tablet_bootstrap.cc:654] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:44.255861 22866 tablet_bootstrap.cc:492] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: Bootstrap starting.
I20260810 03:18:44.257325 22868 tablet_bootstrap.cc:492] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: Bootstrap starting.
I20260810 03:18:44.257808 22866 tablet_bootstrap.cc:654] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:44.258550 22865 tablet_bootstrap.cc:492] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: No bootstrap required, opened a new log
I20260810 03:18:44.258742 22865 ts_tablet_manager.cc:1397] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:18:44.259549 22868 tablet_bootstrap.cc:654] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:44.260094 22865 raft_consensus.cc:348] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.260387 22865 raft_consensus.cc:374] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:44.260500 22865 raft_consensus.cc:729] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef888aa00e35409d964caf74d09d5eff, State: Initialized, Role: FOLLOWER
I20260810 03:18:44.260813 22866 tablet_bootstrap.cc:492] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: No bootstrap required, opened a new log
I20260810 03:18:44.260785 22865 consensus_queue.cc:260] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.260972 22866 ts_tablet_manager.cc:1397] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260810 03:18:44.261511 22865 ts_tablet_manager.cc:1428] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 03:18:44.261314 22872 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.261772 22872 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:44.262243 22868 tablet_bootstrap.cc:492] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: No bootstrap required, opened a new log
I20260810 03:18:44.262414 22868 ts_tablet_manager.cc:1397] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 03:18:44.262614 22866 raft_consensus.cc:348] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.262905 22866 raft_consensus.cc:374] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:44.263025 22866 raft_consensus.cc:729] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b66e0fa6d845dab4da4a135f54115b, State: Initialized, Role: FOLLOWER
I20260810 03:18:44.263343 22866 consensus_queue.cc:260] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.264066 22866 ts_tablet_manager.cc:1428] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:18:44.263928 22873 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.264253 22873 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:44.264169 22868 raft_consensus.cc:348] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.264482 22868 raft_consensus.cc:374] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:44.264633 22868 raft_consensus.cc:729] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e68e0fd34df4ab1bca59d36228cb248, State: Initialized, Role: FOLLOWER
I20260810 03:18:44.264897 22868 consensus_queue.cc:260] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.265430 22874 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.265836 22874 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:44.266417 22868 ts_tablet_manager.cc:1428] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260810 03:18:44.277223 22874 raft_consensus.cc:482] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:44.277469 22874 raft_consensus.cc:504] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.279134 22874 leader_election.cc:288] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521), b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735)
I20260810 03:18:44.286224 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.279751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:47263 (local address 127.19.246.194:41735)
0810 03:18:44.280069 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.280081 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.280357 (+ 276us) server_negotiation.cc:407] Connection header received
0810 03:18:44.280621 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.280627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.280722 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.280795 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:44.281562 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.282520 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.284401 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.285215 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.285409 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.285488 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.285782 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.285920 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260810 03:18:44.286357 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.279551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:41735 (local address 127.19.246.195:47263)
0810 03:18:44.280222 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.280244 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.280402 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.281062 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.281068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.281163 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:44.281391 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.281400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.282694 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.282699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.284233 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.284246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.285421 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.285427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.285611 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.285636 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.285754 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.285902 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"spinlock_wait_cycles":33152,"thread_start_us":367,"threads_started":1}
I20260810 03:18:44.287448 22643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a349b8b4c72840459794644b2919dbc1" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b66e0fa6d845dab4da4a135f54115b" is_pre_election: true
I20260810 03:18:44.287992 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.279551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:45521 (local address 127.19.246.195:56073)
0810 03:18:44.280880 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.280898 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.281118 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.281454 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.281458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.281487 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:44.281988 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.281997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.283203 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.283210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.284737 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.284756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.286949 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.286956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.287153 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.287211 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.287317 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.287540 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"mutex_wait_us":544,"thread_start_us":427,"threads_started":1}
I20260810 03:18:44.288002 22643 raft_consensus.cc:2454] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 0.
I20260810 03:18:44.288792 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.279859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:56073 (local address 127.19.246.193:45521)
0810 03:18:44.280084 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.280090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.280979 (+ 889us) server_negotiation.cc:407] Connection header received
0810 03:18:44.281259 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.281264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.281333 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.281438 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:44.282182 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.283043 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.284915 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.286685 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.286936 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.287063 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.287403 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.287608 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 03:18:44.288764 22718 leader_election.cc:302] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e68e0fd34df4ab1bca59d36228cb248, b0b66e0fa6d845dab4da4a135f54115b; no voters:
I20260810 03:18:44.289449 22494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a349b8b4c72840459794644b2919dbc1" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef888aa00e35409d964caf74d09d5eff" is_pre_election: true
I20260810 03:18:44.289480 22874 raft_consensus.cc:2781] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:44.289747 22494 raft_consensus.cc:2454] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 0.
I20260810 03:18:44.289904 22874 raft_consensus.cc:482] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:44.290094 22874 raft_consensus.cc:3037] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.292718 22874 raft_consensus.cc:504] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.293433 22874 leader_election.cc:288] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 election: Requested vote from peers ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521), b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735)
I20260810 03:18:44.294065 22643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a349b8b4c72840459794644b2919dbc1" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b66e0fa6d845dab4da4a135f54115b"
I20260810 03:18:44.294261 22643 raft_consensus.cc:3037] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.294281 22494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a349b8b4c72840459794644b2919dbc1" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef888aa00e35409d964caf74d09d5eff"
I20260810 03:18:44.294637 22494 raft_consensus.cc:3037] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.296590 22643 raft_consensus.cc:2454] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 1.
I20260810 03:18:44.297260 22718 leader_election.cc:302] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e68e0fd34df4ab1bca59d36228cb248, b0b66e0fa6d845dab4da4a135f54115b; no voters:
I20260810 03:18:44.297675 22874 raft_consensus.cc:2781] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:44.297788 22494 raft_consensus.cc:2454] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 1.
I20260810 03:18:44.298276 22874 raft_consensus.cc:686] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 LEADER]: Becoming Leader. State: Replica: 0e68e0fd34df4ab1bca59d36228cb248, State: Running, Role: LEADER
I20260810 03:18:44.298913 22874 consensus_queue.cc:237] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.302942 22891 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: TxnStatusTablet state changed. Reason: New leader 0e68e0fd34df4ab1bca59d36228cb248. Latest consensus state: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.303308 22891 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:18:44.303928 22893 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:18:44.304311 22893 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:18:44.304236 22324 catalog_manager.cc:5495] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195). New cstate: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } health_report { overall_health: HEALTHY } } }
I20260810 03:18:44.325436 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.318121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.19.246.195:36001)
0810 03:18:44.318355 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.318362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.318593 (+ 231us) server_negotiation.cc:407] Connection header received
0810 03:18:44.318871 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.318879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.319001 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.319181 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:44.319675 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.321051 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.322676 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.323406 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.323581 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.324104 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.324710 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.324881 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.324991 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.325080 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:18:44.325471 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.318136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36001 (local address 127.0.0.1:37338)
0810 03:18:44.318498 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.318514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.318652 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.319247 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.319251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.319262 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:18:44.319520 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.319530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.321200 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.321205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.322472 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.322486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.323610 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.323617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.323813 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.323883 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.324835 (+ 952us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.324865 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.324960 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.325145 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260810 03:18:44.333618 22643 raft_consensus.cc:1261] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Refusing update from remote peer 0e68e0fd34df4ab1bca59d36228cb248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:44.333864 22494 raft_consensus.cc:1261] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Refusing update from remote peer 0e68e0fd34df4ab1bca59d36228cb248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:44.334769 22891 consensus_queue.cc:1035] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:44.335173 22874 consensus_queue.cc:1035] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:44.347412 22873 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: TxnStatusTablet state changed. Reason: New leader 0e68e0fd34df4ab1bca59d36228cb248. Latest consensus state: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.347975 22873 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:44.351548 22896 mvcc.cc:205] Tried to move back new op lower bound from 7316815562059935744 to 7316815561933066240. Current Snapshot: MvccSnapshot[applied={T|T < 7316815562059935744}]
I20260810 03:18:44.353128 22872 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: TxnStatusTablet state changed. Reason: New leader 0e68e0fd34df4ab1bca59d36228cb248. Latest consensus state: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.353539 22872 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:44.355345 22891 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.356376 22891 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:18:44.373008 22872 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.373509 22872 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P ef888aa00e35409d964caf74d09d5eff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:44.379967 22873 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.380425 22873 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P b0b66e0fa6d845dab4da4a135f54115b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:18:44.381467 22905 tablet_replica.cc:439] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } } }
I20260810 03:18:44.381938 22905 tablet_replica.cc:442] T a349b8b4c72840459794644b2919dbc1 P 0e68e0fd34df4ab1bca59d36228cb248: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:18:44.392627 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.379437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.19.246.194:41735)
0810 03:18:44.379659 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.379667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.385351 (+ 5684us) server_negotiation.cc:407] Connection header received
0810 03:18:44.385619 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.385623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.385742 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.385883 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:44.386560 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.387507 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.389854 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.390310 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.390423 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.391044 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.391753 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.391952 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.392125 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.392245 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:18:44.413182 22913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.406479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36001 (local address 127.19.246.194:45791)
0810 03:18:44.407268 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.407292 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.407531 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.407946 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.407949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.407960 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:18:44.408239 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.408247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.409718 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.409736 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.410879 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.410892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.412009 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.412014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.412136 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.412201 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.412683 (+ 482us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.412692 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.412761 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.412877 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":399,"threads_started":1}
I20260810 03:18:44.413198 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.406474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:45791 (local address 127.19.246.195:36001)
0810 03:18:44.406689 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.406697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.407434 (+ 737us) server_negotiation.cc:407] Connection header received
0810 03:18:44.407684 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.407690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.407810 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.407920 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:44.408388 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.409554 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.411034 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.411814 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.411964 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.412419 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.412577 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.412699 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.412813 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.412900 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:18:44.423774 22493 raft_consensus.cc:1261] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Refusing update from remote peer b0b66e0fa6d845dab4da4a135f54115b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:44.423801 22792 raft_consensus.cc:1261] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Refusing update from remote peer b0b66e0fa6d845dab4da4a135f54115b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:44.424577 22873 consensus_queue.cc:1035] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:44.425228 22877 consensus_queue.cc:1035] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:44.432526 22897 mvcc.cc:205] Tried to move back new op lower bound from 7316815562431619072 to 7316815561315713024. Current Snapshot: MvccSnapshot[applied={T|T < 7316815562431619072}]
I20260810 03:18:44.433825 22898 mvcc.cc:205] Tried to move back new op lower bound from 7316815562431619072 to 7316815561315713024. Current Snapshot: MvccSnapshot[applied={T|T < 7316815562431619072}]
I20260810 03:18:44.465174 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.457819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:49657 (local address 127.19.246.194:41735)
0810 03:18:44.457999 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.458006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.458402 (+ 396us) server_negotiation.cc:407] Connection header received
0810 03:18:44.458664 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.458671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.458840 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.458953 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:44.459686 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.460493 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.462808 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.463509 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.463687 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.464250 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.464370 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.464482 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.464692 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.464789 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:18:44.465572 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.457435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:41735 (local address 127.19.246.195:49657)
0810 03:18:44.458265 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.458290 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.458497 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.458975 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.458981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.458997 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:18:44.459498 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.459511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.460666 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.460672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.462584 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.462603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.463743 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.463754 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.463930 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.464005 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.464517 (+ 512us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.464531 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.464657 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.464822 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":581,"threads_started":1}
I20260810 03:18:44.516629 22323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:44.516943 22323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:44.521694 22323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: Sending DeleteTablet for 3 replicas of tablet 972b61b4bd874401b4df89339050783b
I20260810 03:18:44.523020 22623 tablet_service.cc:1514] Processing DeleteTablet for tablet 972b61b4bd874401b4df89339050783b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:44 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260810 03:18:44.523150 22770 tablet_service.cc:1514] Processing DeleteTablet for tablet 972b61b4bd874401b4df89339050783b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:44 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:37326
I20260810 03:18:44.524641 22473 tablet_service.cc:1514] Processing DeleteTablet for tablet 972b61b4bd874401b4df89339050783b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:44 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38868
I20260810 03:18:44.531971 22924 tablet_replica.cc:330] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: stopping tablet replica
I20260810 03:18:44.532539 22924 raft_consensus.cc:2229] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:44.532974 22924 raft_consensus.cc:2258] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:44.533780 22923 tablet_replica.cc:330] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: stopping tablet replica
I20260810 03:18:44.534233 22923 raft_consensus.cc:2229] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:18:44.534116 22323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:18:44.534955 22923 raft_consensus.cc:2258] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 03:18:44.535287 22323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:44.536849 22924 ts_tablet_manager.cc:1905] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:44.544023 22473 tablet_service.cc:1467] Processing CreateTablet for tablet 2b511d5419a24b1483dcd40ed48cb23c (DEFAULT_TABLE table=test-table [id=7bbcab52319c4975a6d0c470933b256f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:44.544348 22769 tablet_service.cc:1467] Processing CreateTablet for tablet 2b511d5419a24b1483dcd40ed48cb23c (DEFAULT_TABLE table=test-table [id=7bbcab52319c4975a6d0c470933b256f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:44.544575 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b511d5419a24b1483dcd40ed48cb23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:44.544920 22769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b511d5419a24b1483dcd40ed48cb23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:44.545390 22923 ts_tablet_manager.cc:1905] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:44.548565 22621 tablet_service.cc:1467] Processing CreateTablet for tablet 2b511d5419a24b1483dcd40ed48cb23c (DEFAULT_TABLE table=test-table [id=7bbcab52319c4975a6d0c470933b256f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:44.549219 22621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b511d5419a24b1483dcd40ed48cb23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:44.552273 22865 tablet_bootstrap.cc:492] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: Bootstrap starting.
I20260810 03:18:44.554200 22865 tablet_bootstrap.cc:654] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:44.554904 22868 tablet_bootstrap.cc:492] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: Bootstrap starting.
I20260810 03:18:44.556102 22866 tablet_bootstrap.cc:492] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: Bootstrap starting.
I20260810 03:18:44.556607 22868 tablet_bootstrap.cc:654] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:44.557938 22866 tablet_bootstrap.cc:654] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:44.569433 22865 tablet_bootstrap.cc:492] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: No bootstrap required, opened a new log
I20260810 03:18:44.569665 22865 ts_tablet_manager.cc:1397] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260810 03:18:44.570466 22921 tablet_replica.cc:330] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: stopping tablet replica
I20260810 03:18:44.570552 22865 raft_consensus.cc:348] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.570853 22865 raft_consensus.cc:374] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:44.570952 22865 raft_consensus.cc:729] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef888aa00e35409d964caf74d09d5eff, State: Initialized, Role: FOLLOWER
I20260810 03:18:44.571344 22865 consensus_queue.cc:260] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.573776 22921 raft_consensus.cc:2229] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:44.574604 22921 raft_consensus.cc:2258] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:44.575102 22865 ts_tablet_manager.cc:1428] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 03:18:44.578187 22868 tablet_bootstrap.cc:492] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: No bootstrap required, opened a new log
I20260810 03:18:44.578709 22866 tablet_bootstrap.cc:492] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: No bootstrap required, opened a new log
I20260810 03:18:44.578706 22868 ts_tablet_manager.cc:1397] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.011s
I20260810 03:18:44.578910 22866 ts_tablet_manager.cc:1397] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.001s
I20260810 03:18:44.579799 22866 raft_consensus.cc:348] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.579809 22868 raft_consensus.cc:348] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.580075 22866 raft_consensus.cc:374] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:44.580158 22866 raft_consensus.cc:729] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b66e0fa6d845dab4da4a135f54115b, State: Initialized, Role: FOLLOWER
I20260810 03:18:44.580307 22868 raft_consensus.cc:374] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:44.580456 22868 raft_consensus.cc:729] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e68e0fd34df4ab1bca59d36228cb248, State: Initialized, Role: FOLLOWER
I20260810 03:18:44.580487 22866 consensus_queue.cc:260] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.580892 22868 consensus_queue.cc:260] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.582859 22868 ts_tablet_manager.cc:1428] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 03:18:44.579818 22921 ts_tablet_manager.cc:1905] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:44.584434 22866 ts_tablet_manager.cc:1428] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 03:18:44.589327 22924 ts_tablet_manager.cc:1918] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:44.589545 22924 log.cc:1196] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/972b61b4bd874401b4df89339050783b
I20260810 03:18:44.590194 22924 ts_tablet_manager.cc:1939] T 972b61b4bd874401b4df89339050783b P ef888aa00e35409d964caf74d09d5eff: Deleting consensus metadata
I20260810 03:18:44.590971 22923 ts_tablet_manager.cc:1918] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:44.591178 22923 log.cc:1196] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/972b61b4bd874401b4df89339050783b
I20260810 03:18:44.591807 22923 ts_tablet_manager.cc:1939] T 972b61b4bd874401b4df89339050783b P b0b66e0fa6d845dab4da4a135f54115b: Deleting consensus metadata
I20260810 03:18:44.592469 22306 catalog_manager.cc:4841] TS ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521): tablet 972b61b4bd874401b4df89339050783b (table test-table [id=bdfdef0edf6b4c29ba52df3d46e19d03]) successfully deleted
I20260810 03:18:44.594035 22309 catalog_manager.cc:4841] TS b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735): tablet 972b61b4bd874401b4df89339050783b (table test-table [id=bdfdef0edf6b4c29ba52df3d46e19d03]) successfully deleted
I20260810 03:18:44.595924 22921 ts_tablet_manager.cc:1918] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:44.596155 22921 log.cc:1196] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/972b61b4bd874401b4df89339050783b
I20260810 03:18:44.596679 22921 ts_tablet_manager.cc:1939] T 972b61b4bd874401b4df89339050783b P 0e68e0fd34df4ab1bca59d36228cb248: Deleting consensus metadata
I20260810 03:18:44.598300 22310 catalog_manager.cc:4841] TS 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001): tablet 972b61b4bd874401b4df89339050783b (table test-table [id=bdfdef0edf6b4c29ba52df3d46e19d03]) successfully deleted
I20260810 03:18:44.800364 22874 raft_consensus.cc:482] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:44.800659 22874 raft_consensus.cc:504] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.801585 22874 leader_election.cc:288] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521), b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735)
I20260810 03:18:44.802935 22493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b511d5419a24b1483dcd40ed48cb23c" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef888aa00e35409d964caf74d09d5eff" is_pre_election: true
I20260810 03:18:44.803617 22643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b511d5419a24b1483dcd40ed48cb23c" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b66e0fa6d845dab4da4a135f54115b" is_pre_election: true
I20260810 03:18:44.803905 22643 raft_consensus.cc:2454] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 0.
I20260810 03:18:44.804636 22718 leader_election.cc:302] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e68e0fd34df4ab1bca59d36228cb248, b0b66e0fa6d845dab4da4a135f54115b; no voters:
I20260810 03:18:44.803262 22493 raft_consensus.cc:2454] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 0.
I20260810 03:18:44.805667 22874 raft_consensus.cc:2781] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:44.805819 22874 raft_consensus.cc:482] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:44.805967 22874 raft_consensus.cc:3037] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.809906 22874 raft_consensus.cc:504] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.811422 22493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b511d5419a24b1483dcd40ed48cb23c" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef888aa00e35409d964caf74d09d5eff"
I20260810 03:18:44.811751 22493 raft_consensus.cc:3037] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.815791 22493 raft_consensus.cc:2454] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 1.
I20260810 03:18:44.816490 22716 leader_election.cc:302] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e68e0fd34df4ab1bca59d36228cb248, ef888aa00e35409d964caf74d09d5eff; no voters:
I20260810 03:18:44.816949 22891 raft_consensus.cc:2781] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:44.817174 22891 raft_consensus.cc:686] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 LEADER]: Becoming Leader. State: Replica: 0e68e0fd34df4ab1bca59d36228cb248, State: Running, Role: LEADER
I20260810 03:18:44.817513 22891 consensus_queue.cc:237] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:44.821871 22874 leader_election.cc:288] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 election: Requested vote from peers ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521), b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735)
I20260810 03:18:44.821974 22643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b511d5419a24b1483dcd40ed48cb23c" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b66e0fa6d845dab4da4a135f54115b"
I20260810 03:18:44.822377 22643 raft_consensus.cc:3037] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:44.828065 22643 raft_consensus.cc:2454] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 1.
I20260810 03:18:44.831836 22323 catalog_manager.cc:5495] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195). New cstate: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } health_report { overall_health: HEALTHY } } }
I20260810 03:18:44.882030 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.876866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.19.246.195:36001)
0810 03:18:44.877035 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.877041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.877155 (+ 114us) server_negotiation.cc:407] Connection header received
0810 03:18:44.877361 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.877367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.877442 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.877544 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:44.877917 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.878727 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.880124 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.880659 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.880816 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.881205 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.881327 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.881430 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.881587 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.881678 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:18:44.896415 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.890970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:50571 (local address 127.19.246.195:36001)
0810 03:18:44.891537 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:44.891540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:44.891551 (+ 11us) server_negotiation.cc:407] Connection header received
0810 03:18:44.891591 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:44.891594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:44.891662 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:44.891767 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:44.892236 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.893069 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.894613 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.895038 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.895198 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.895625 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.895750 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.895848 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:44.895954 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 03:18:44.896048 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":484}
I20260810 03:18:44.896435 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:44.891001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36001 (local address 127.19.246.195:50571)
0810 03:18:44.891187 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:44.891204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:44.891311 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:44.891781 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:44.891785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:44.891795 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:18:44.892027 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.892036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.893217 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.893224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.894466 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:44.894479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:44.895196 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:44.895202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:44.895412 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:44.895465 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:18:44.895847 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:18:44.895859 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:18:44.895941 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:18:44.896139 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":45}
I20260810 03:18:44.907691 22494 raft_consensus.cc:1261] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Refusing update from remote peer 0e68e0fd34df4ab1bca59d36228cb248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:44.907694 22643 raft_consensus.cc:1261] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Refusing update from remote peer 0e68e0fd34df4ab1bca59d36228cb248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:44.908448 22874 consensus_queue.cc:1035] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:44.908806 22891 consensus_queue.cc:1035] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:18:44.952666 22896 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 03:18:46.580201 22323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:46.580528 22323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:46.585719 22323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: Sending DeleteTablet for 3 replicas of tablet 2b511d5419a24b1483dcd40ed48cb23c
I20260810 03:18:46.586799 22473 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b511d5419a24b1483dcd40ed48cb23c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:46 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38868
I20260810 03:18:46.586799 22623 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b511d5419a24b1483dcd40ed48cb23c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:46 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260810 03:18:46.586860 22769 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b511d5419a24b1483dcd40ed48cb23c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:46 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:37326
I20260810 03:18:46.601315 22956 tablet_replica.cc:330] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: stopping tablet replica
I20260810 03:18:46.601740 22952 tablet_replica.cc:330] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: stopping tablet replica
I20260810 03:18:46.601691 22956 raft_consensus.cc:2229] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:18:46.602057 22952 raft_consensus.cc:2229] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:46.602416 22956 raft_consensus.cc:2258] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:46.602435 22952 raft_consensus.cc:2258] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:46.603747 22954 tablet_replica.cc:330] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: stopping tablet replica
I20260810 03:18:46.604009 22956 ts_tablet_manager.cc:1905] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:46.603996 22954 raft_consensus.cc:2229] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:46.604293 22954 raft_consensus.cc:2258] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:46.604797 22952 ts_tablet_manager.cc:1905] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:46.605305 22954 ts_tablet_manager.cc:1905] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:46.612762 22954 ts_tablet_manager.cc:1918] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:18:46.612762 22952 ts_tablet_manager.cc:1918] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:18:46.612949 22954 log.cc:1196] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/2b511d5419a24b1483dcd40ed48cb23c
I20260810 03:18:46.613137 22952 log.cc:1196] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/2b511d5419a24b1483dcd40ed48cb23c
I20260810 03:18:46.613508 22954 ts_tablet_manager.cc:1939] T 2b511d5419a24b1483dcd40ed48cb23c P b0b66e0fa6d845dab4da4a135f54115b: Deleting consensus metadata
I20260810 03:18:46.613780 22952 ts_tablet_manager.cc:1939] T 2b511d5419a24b1483dcd40ed48cb23c P ef888aa00e35409d964caf74d09d5eff: Deleting consensus metadata
I20260810 03:18:46.614167 22956 ts_tablet_manager.cc:1918] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:18:46.614455 22956 log.cc:1196] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/2b511d5419a24b1483dcd40ed48cb23c
I20260810 03:18:46.615264 22309 catalog_manager.cc:4841] TS b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735): tablet 2b511d5419a24b1483dcd40ed48cb23c (table test-table [id=7bbcab52319c4975a6d0c470933b256f]) successfully deleted
I20260810 03:18:46.615134 22956 ts_tablet_manager.cc:1939] T 2b511d5419a24b1483dcd40ed48cb23c P 0e68e0fd34df4ab1bca59d36228cb248: Deleting consensus metadata
I20260810 03:18:46.615500 22306 catalog_manager.cc:4841] TS ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521): tablet 2b511d5419a24b1483dcd40ed48cb23c (table test-table [id=7bbcab52319c4975a6d0c470933b256f]) successfully deleted
I20260810 03:18:46.617005 22310 catalog_manager.cc:4841] TS 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001): tablet 2b511d5419a24b1483dcd40ed48cb23c (table test-table [id=7bbcab52319c4975a6d0c470933b256f]) successfully deleted
Aug 10 03:18:46 dist-test-slave-btw8 krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:18:46 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:46.672317 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.648195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.19.246.254:39681)
0810 03:18:46.648808 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:46.648812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:46.648834 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:18:46.648891 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:46.648895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:46.648969 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:46.649112 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:46.651372 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.652855 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.655155 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.655997 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.656173 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.662023 (+ 5850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:46.662031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:46.662036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:46.662092 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:46.666967 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:46.668083 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:46.668091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:46.668096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:46.668300 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:46.668945 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:46.668951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:46.668957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:46.669482 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:46.669797 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:46.671433 (+ 1636us) server_negotiation.cc:299] Negotiation successful
0810 03:18:46.671797 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":318,"threads_started":1}
I20260810 03:18:46.677611 22323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:18:46.678457 22323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:46.687655 22473 tablet_service.cc:1467] Processing CreateTablet for tablet 129bccbffae94bf6a514e71d20600acf (DEFAULT_TABLE table=test-table [id=fefde700e90b4ea08557d7c9515c41c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:46.688256 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129bccbffae94bf6a514e71d20600acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:46.688522 22769 tablet_service.cc:1467] Processing CreateTablet for tablet 129bccbffae94bf6a514e71d20600acf (DEFAULT_TABLE table=test-table [id=fefde700e90b4ea08557d7c9515c41c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:46.689285 22769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129bccbffae94bf6a514e71d20600acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:46.693359 22623 tablet_service.cc:1467] Processing CreateTablet for tablet 129bccbffae94bf6a514e71d20600acf (DEFAULT_TABLE table=test-table [id=fefde700e90b4ea08557d7c9515c41c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:46.693907 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129bccbffae94bf6a514e71d20600acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:46.696362 22966 tablet_bootstrap.cc:492] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: Bootstrap starting.
I20260810 03:18:46.698309 22967 tablet_bootstrap.cc:492] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: Bootstrap starting.
I20260810 03:18:46.698343 22966 tablet_bootstrap.cc:654] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:46.700362 22967 tablet_bootstrap.cc:654] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:46.700783 22966 tablet_bootstrap.cc:492] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: No bootstrap required, opened a new log
I20260810 03:18:46.700956 22966 ts_tablet_manager.cc:1397] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 03:18:46.701889 22966 raft_consensus.cc:348] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.702222 22966 raft_consensus.cc:374] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:46.702324 22966 raft_consensus.cc:729] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef888aa00e35409d964caf74d09d5eff, State: Initialized, Role: FOLLOWER
I20260810 03:18:46.702765 22966 consensus_queue.cc:260] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.703795 22967 tablet_bootstrap.cc:492] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: No bootstrap required, opened a new log
I20260810 03:18:46.703960 22967 ts_tablet_manager.cc:1397] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:18:46.703974 22966 ts_tablet_manager.cc:1428] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 03:18:46.705538 22967 raft_consensus.cc:348] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.705830 22967 raft_consensus.cc:374] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:46.705979 22967 raft_consensus.cc:729] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e68e0fd34df4ab1bca59d36228cb248, State: Initialized, Role: FOLLOWER
I20260810 03:18:46.706269 22967 consensus_queue.cc:260] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.709082 22967 ts_tablet_manager.cc:1428] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 03:18:46.713887 22971 tablet_bootstrap.cc:492] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: Bootstrap starting.
I20260810 03:18:46.716014 22971 tablet_bootstrap.cc:654] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:46.719105 22971 tablet_bootstrap.cc:492] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: No bootstrap required, opened a new log
I20260810 03:18:46.719729 22971 ts_tablet_manager.cc:1397] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:18:46.720646 22971 raft_consensus.cc:348] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.720978 22971 raft_consensus.cc:374] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:46.721061 22971 raft_consensus.cc:729] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b66e0fa6d845dab4da4a135f54115b, State: Initialized, Role: FOLLOWER
I20260810 03:18:46.721340 22971 consensus_queue.cc:260] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.722188 22971 ts_tablet_manager.cc:1428] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:18:46.832819 22970 raft_consensus.cc:482] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:46.833138 22970 raft_consensus.cc:504] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.834851 22970 leader_election.cc:288] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735), 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001)
I20260810 03:18:46.857576 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.835514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:49035 (local address 127.19.246.195:36001)
0810 03:18:46.848052 (+ 12538us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:46.848059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:46.848852 (+ 793us) server_negotiation.cc:407] Connection header received
0810 03:18:46.849054 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:46.849060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:46.849148 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:46.849267 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:46.850565 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.851951 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.855422 (+ 3471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.856320 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.856533 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.856642 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:46.857027 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 03:18:46.857204 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12377,"thread_start_us":11686,"threads_started":1}
I20260810 03:18:46.858350 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.835072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:41735 (local address 127.19.246.193:57533)
0810 03:18:46.848074 (+ 13002us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:46.848101 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:46.848331 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:46.849598 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:46.849604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:46.849652 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:46.850254 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:46.850264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.852168 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.852182 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:46.855134 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:46.855155 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.856748 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.856756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:46.856994 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.857032 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:18:46.857137 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 03:18:46.857279 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12788,"mutex_wait_us":280,"spinlock_wait_cycles":896,"thread_start_us":13137,"threads_started":1}
I20260810 03:18:46.858494 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.835238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36001 (local address 127.19.246.193:49035)
0810 03:18:46.848680 (+ 13442us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:46.848706 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:46.848900 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:46.849598 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:46.849605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:46.849660 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:18:46.850224 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:46.850245 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.852191 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.852195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:46.855134 (+ 2939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:46.855155 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.856558 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.856566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:46.856838 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.856868 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:18:46.856960 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:18:46.857079 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13242,"mutex_wait_us":171,"spinlock_wait_cycles":6272,"thread_start_us":12388,"threads_started":1}
I20260810 03:18:46.858690 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.835442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:57533 (local address 127.19.246.194:41735)
0810 03:18:46.849023 (+ 13581us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:46.849029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:46.849055 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:18:46.849124 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:46.849130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:46.849225 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:46.849341 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:18:46.850577 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.851954 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.855431 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.856408 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.856632 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.856715 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:46.857695 (+ 980us) server_negotiation.cc:299] Negotiation successful
0810 03:18:46.858100 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13402,"thread_start_us":13198,"threads_started":1}
I20260810 03:18:46.859480 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129bccbffae94bf6a514e71d20600acf" candidate_uuid: "ef888aa00e35409d964caf74d09d5eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e68e0fd34df4ab1bca59d36228cb248" is_pre_election: true
I20260810 03:18:46.859495 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129bccbffae94bf6a514e71d20600acf" candidate_uuid: "ef888aa00e35409d964caf74d09d5eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b66e0fa6d845dab4da4a135f54115b" is_pre_election: true
I20260810 03:18:46.859752 22792 raft_consensus.cc:2454] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef888aa00e35409d964caf74d09d5eff in term 0.
I20260810 03:18:46.859767 22642 raft_consensus.cc:2454] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef888aa00e35409d964caf74d09d5eff in term 0.
I20260810 03:18:46.860427 22419 leader_election.cc:302] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e68e0fd34df4ab1bca59d36228cb248, b0b66e0fa6d845dab4da4a135f54115b, ef888aa00e35409d964caf74d09d5eff; no voters:
I20260810 03:18:46.860953 22970 raft_consensus.cc:2781] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:46.861106 22970 raft_consensus.cc:482] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:46.861217 22970 raft_consensus.cc:3037] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:46.863945 22970 raft_consensus.cc:504] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.864768 22970 leader_election.cc:288] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [CANDIDATE]: Term 1 election: Requested vote from peers b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735), 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001)
I20260810 03:18:46.865685 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129bccbffae94bf6a514e71d20600acf" candidate_uuid: "ef888aa00e35409d964caf74d09d5eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e68e0fd34df4ab1bca59d36228cb248"
I20260810 03:18:46.865638 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129bccbffae94bf6a514e71d20600acf" candidate_uuid: "ef888aa00e35409d964caf74d09d5eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b66e0fa6d845dab4da4a135f54115b"
I20260810 03:18:46.865940 22792 raft_consensus.cc:3037] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:46.865949 22642 raft_consensus.cc:3037] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:46.869258 22792 raft_consensus.cc:2454] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef888aa00e35409d964caf74d09d5eff in term 1.
I20260810 03:18:46.869259 22642 raft_consensus.cc:2454] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef888aa00e35409d964caf74d09d5eff in term 1.
I20260810 03:18:46.869889 22420 leader_election.cc:302] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e68e0fd34df4ab1bca59d36228cb248, b0b66e0fa6d845dab4da4a135f54115b, ef888aa00e35409d964caf74d09d5eff; no voters:
I20260810 03:18:46.870409 22970 raft_consensus.cc:2781] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:46.870954 22970 raft_consensus.cc:686] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 1 LEADER]: Becoming Leader. State: Replica: ef888aa00e35409d964caf74d09d5eff, State: Running, Role: LEADER
I20260810 03:18:46.871487 22970 consensus_queue.cc:237] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:46.875787 22323 catalog_manager.cc:5495] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff reported cstate change: term changed from 0 to 1, leader changed from <none> to ef888aa00e35409d964caf74d09d5eff (127.19.246.193). New cstate: current_term: 1 leader_uuid: "ef888aa00e35409d964caf74d09d5eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } health_report { overall_health: UNKNOWN } } }
I20260810 03:18:46.926220 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.918232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.19.246.193:45521)
0810 03:18:46.919040 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:46.919045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:46.919066 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:18:46.919313 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:46.919319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:46.919465 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:46.919637 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:46.920255 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.921082 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.922850 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.923725 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.923853 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.924600 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:46.925336 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:46.925484 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:46.925707 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 03:18:46.925800 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":549,"threads_started":1}
I20260810 03:18:46.946036 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.939901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:36289 (local address 127.19.246.195:36001)
0810 03:18:46.940175 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:46.940188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:46.940648 (+ 460us) server_negotiation.cc:407] Connection header received
0810 03:18:46.940901 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:46.940905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:46.940973 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:46.941075 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:46.941585 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.942457 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.944067 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.944602 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.944735 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.945252 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:46.945392 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:46.945481 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:46.945675 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 03:18:46.945761 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 03:18:46.946101 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.939648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36001 (local address 127.19.246.193:36289)
0810 03:18:46.940526 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:46.940545 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:46.940741 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:46.941094 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:46.941098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:46.941110 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:18:46.941431 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:46.941440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.942613 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.942618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:46.943913 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:46.943926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.944817 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.944821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:46.944974 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.945039 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:18:46.945537 (+ 498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:18:46.945550 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 03:18:46.945635 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 03:18:46.945789 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":500,"threads_started":1}
I20260810 03:18:46.956123 22792 raft_consensus.cc:1261] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Refusing update from remote peer ef888aa00e35409d964caf74d09d5eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:46.956153 22642 raft_consensus.cc:1261] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Refusing update from remote peer ef888aa00e35409d964caf74d09d5eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:46.956998 22978 consensus_queue.cc:1035] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:46.957406 22970 consensus_queue.cc:1035] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:46.999583 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.990997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:39689 (local address 127.19.246.193:45521)
0810 03:18:46.991185 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:46.991192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:46.992037 (+ 845us) server_negotiation.cc:407] Connection header received
0810 03:18:46.992277 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:46.992284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:46.992390 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:46.992483 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:46.993121 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.994209 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.996532 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.997164 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.997289 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.998037 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:46.998163 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:46.998273 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:46.998661 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 03:18:46.998894 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:18:46.999625 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:46.991134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:45521 (local address 127.19.246.195:39689)
0810 03:18:46.991912 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:46.991939 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:46.992101 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:46.992548 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:46.992554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:46.992569 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:18:46.992920 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:46.992929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.994578 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.994586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:46.996282 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:46.996301 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:46.997405 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:46.997410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:46.997610 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:46.997654 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:18:46.998342 (+ 688us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:18:46.998354 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:18:46.998432 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:18:46.998699 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":491,"threads_started":1}
I20260810 03:18:47.042836 22324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:47.043092 22324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:47.046494 22324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: Sending DeleteTablet for 3 replicas of tablet 129bccbffae94bf6a514e71d20600acf
I20260810 03:18:47.047567 22623 tablet_service.cc:1514] Processing DeleteTablet for tablet 129bccbffae94bf6a514e71d20600acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:47 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260810 03:18:47.047564 22473 tablet_service.cc:1514] Processing DeleteTablet for tablet 129bccbffae94bf6a514e71d20600acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:47 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38868
I20260810 03:18:47.047899 22952 tablet_replica.cc:330] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: stopping tablet replica
I20260810 03:18:47.047942 22954 tablet_replica.cc:330] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: stopping tablet replica
I20260810 03:18:47.048111 22952 raft_consensus.cc:2229] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:18:47.048151 22954 raft_consensus.cc:2229] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:47.048154 22769 tablet_service.cc:1514] Processing DeleteTablet for tablet 129bccbffae94bf6a514e71d20600acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:47 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:37326
I20260810 03:18:47.048357 22954 raft_consensus.cc:2258] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:47.048454 22956 tablet_replica.cc:330] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: stopping tablet replica
I20260810 03:18:47.048658 22956 raft_consensus.cc:2229] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:47.048703 22952 raft_consensus.cc:2258] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:47.048861 22956 raft_consensus.cc:2258] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:47.049939 22952 ts_tablet_manager.cc:1905] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:47.050132 22956 ts_tablet_manager.cc:1905] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:47.050582 22324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:18:47.051733 22324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:18:47.057287 22954 ts_tablet_manager.cc:1905] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:47.061172 22956 ts_tablet_manager.cc:1918] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:47.061839 22956 log.cc:1196] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/129bccbffae94bf6a514e71d20600acf
I20260810 03:18:47.062548 22956 ts_tablet_manager.cc:1939] T 129bccbffae94bf6a514e71d20600acf P 0e68e0fd34df4ab1bca59d36228cb248: Deleting consensus metadata
I20260810 03:18:47.064088 22310 catalog_manager.cc:4841] TS 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001): tablet 129bccbffae94bf6a514e71d20600acf (table test-table [id=fefde700e90b4ea08557d7c9515c41c8]) successfully deleted
I20260810 03:18:47.064096 22473 tablet_service.cc:1467] Processing CreateTablet for tablet 3aeb4caf9dd2470e996edf28ca5838a0 (DEFAULT_TABLE table=test-table [id=5e991cc7186d4b29a642758a0f425da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:47.064131 22769 tablet_service.cc:1467] Processing CreateTablet for tablet 3aeb4caf9dd2470e996edf28ca5838a0 (DEFAULT_TABLE table=test-table [id=5e991cc7186d4b29a642758a0f425da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:47.064642 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aeb4caf9dd2470e996edf28ca5838a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:47.064687 22769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aeb4caf9dd2470e996edf28ca5838a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:47.066778 22954 ts_tablet_manager.cc:1918] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:47.067037 22954 log.cc:1196] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/129bccbffae94bf6a514e71d20600acf
I20260810 03:18:47.067627 22954 ts_tablet_manager.cc:1939] T 129bccbffae94bf6a514e71d20600acf P b0b66e0fa6d845dab4da4a135f54115b: Deleting consensus metadata
I20260810 03:18:47.067970 22952 ts_tablet_manager.cc:1918] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:18:47.068181 22952 log.cc:1196] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/129bccbffae94bf6a514e71d20600acf
I20260810 03:18:47.068626 22952 ts_tablet_manager.cc:1939] T 129bccbffae94bf6a514e71d20600acf P ef888aa00e35409d964caf74d09d5eff: Deleting consensus metadata
I20260810 03:18:47.069574 22309 catalog_manager.cc:4841] TS b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735): tablet 129bccbffae94bf6a514e71d20600acf (table test-table [id=fefde700e90b4ea08557d7c9515c41c8]) successfully deleted
I20260810 03:18:47.069862 22306 catalog_manager.cc:4841] TS ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521): tablet 129bccbffae94bf6a514e71d20600acf (table test-table [id=fefde700e90b4ea08557d7c9515c41c8]) successfully deleted
I20260810 03:18:47.070464 22623 tablet_service.cc:1467] Processing CreateTablet for tablet 3aeb4caf9dd2470e996edf28ca5838a0 (DEFAULT_TABLE table=test-table [id=5e991cc7186d4b29a642758a0f425da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:18:47.070967 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aeb4caf9dd2470e996edf28ca5838a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:47.072275 22967 tablet_bootstrap.cc:492] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: Bootstrap starting.
I20260810 03:18:47.074152 22967 tablet_bootstrap.cc:654] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:47.076107 22966 tablet_bootstrap.cc:492] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: Bootstrap starting.
I20260810 03:18:47.079393 22966 tablet_bootstrap.cc:654] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:47.081135 22971 tablet_bootstrap.cc:492] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: Bootstrap starting.
I20260810 03:18:47.084093 22971 tablet_bootstrap.cc:654] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:47.088836 22971 tablet_bootstrap.cc:492] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: No bootstrap required, opened a new log
I20260810 03:18:47.089098 22971 ts_tablet_manager.cc:1397] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 03:18:47.089120 22967 tablet_bootstrap.cc:492] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: No bootstrap required, opened a new log
I20260810 03:18:47.089504 22967 ts_tablet_manager.cc:1397] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20260810 03:18:47.090484 22967 raft_consensus.cc:348] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.090392 22971 raft_consensus.cc:348] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.090816 22971 raft_consensus.cc:374] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:47.090938 22971 raft_consensus.cc:729] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b66e0fa6d845dab4da4a135f54115b, State: Initialized, Role: FOLLOWER
I20260810 03:18:47.090945 22967 raft_consensus.cc:374] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:47.091174 22967 raft_consensus.cc:729] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e68e0fd34df4ab1bca59d36228cb248, State: Initialized, Role: FOLLOWER
I20260810 03:18:47.091607 22967 consensus_queue.cc:260] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.091635 22971 consensus_queue.cc:260] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.093154 22967 ts_tablet_manager.cc:1428] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:18:47.094141 22971 ts_tablet_manager.cc:1428] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 03:18:47.098001 22966 tablet_bootstrap.cc:492] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: No bootstrap required, opened a new log
I20260810 03:18:47.098376 22966 ts_tablet_manager.cc:1397] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.010s
I20260810 03:18:47.099344 22966 raft_consensus.cc:348] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.099722 22966 raft_consensus.cc:374] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:47.099889 22966 raft_consensus.cc:729] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef888aa00e35409d964caf74d09d5eff, State: Initialized, Role: FOLLOWER
I20260810 03:18:47.100179 22966 consensus_queue.cc:260] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.101027 22947 raft_consensus.cc:482] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:18:47.101266 22947 raft_consensus.cc:504] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.102038 22947 leader_election.cc:288] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521), b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735)
I20260810 03:18:47.100852 22966 ts_tablet_manager.cc:1428] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:18:47.102627 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aeb4caf9dd2470e996edf28ca5838a0" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b66e0fa6d845dab4da4a135f54115b" is_pre_election: true
I20260810 03:18:47.102945 22642 raft_consensus.cc:2454] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 0.
I20260810 03:18:47.103698 22494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aeb4caf9dd2470e996edf28ca5838a0" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef888aa00e35409d964caf74d09d5eff" is_pre_election: true
I20260810 03:18:47.103837 22718 leader_election.cc:302] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e68e0fd34df4ab1bca59d36228cb248, b0b66e0fa6d845dab4da4a135f54115b; no voters:
I20260810 03:18:47.103936 22494 raft_consensus.cc:2454] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 0.
I20260810 03:18:47.104136 22947 raft_consensus.cc:2781] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:18:47.104256 22947 raft_consensus.cc:482] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:18:47.104354 22947 raft_consensus.cc:3037] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:47.107658 22947 raft_consensus.cc:504] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.108482 22947 leader_election.cc:288] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 election: Requested vote from peers ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521), b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735)
I20260810 03:18:47.109200 22494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aeb4caf9dd2470e996edf28ca5838a0" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef888aa00e35409d964caf74d09d5eff"
I20260810 03:18:47.109362 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aeb4caf9dd2470e996edf28ca5838a0" candidate_uuid: "0e68e0fd34df4ab1bca59d36228cb248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b66e0fa6d845dab4da4a135f54115b"
I20260810 03:18:47.109491 22494 raft_consensus.cc:3037] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:47.109706 22642 raft_consensus.cc:3037] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:47.113503 22642 raft_consensus.cc:2454] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 1.
I20260810 03:18:47.113511 22494 raft_consensus.cc:2454] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e68e0fd34df4ab1bca59d36228cb248 in term 1.
I20260810 03:18:47.114096 22718 leader_election.cc:302] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e68e0fd34df4ab1bca59d36228cb248, b0b66e0fa6d845dab4da4a135f54115b; no voters:
I20260810 03:18:47.114544 22947 raft_consensus.cc:2781] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:47.114828 22947 raft_consensus.cc:686] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 LEADER]: Becoming Leader. State: Replica: 0e68e0fd34df4ab1bca59d36228cb248, State: Running, Role: LEADER
I20260810 03:18:47.115161 22947 consensus_queue.cc:237] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } }
I20260810 03:18:47.118777 22323 catalog_manager.cc:5495] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195). New cstate: current_term: 1 leader_uuid: "0e68e0fd34df4ab1bca59d36228cb248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e68e0fd34df4ab1bca59d36228cb248" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36001 } health_report { overall_health: HEALTHY } } }
I20260810 03:18:47.152585 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:47.145432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.19.246.195:36001)
0810 03:18:47.145621 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:47.145626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:47.145969 (+ 343us) server_negotiation.cc:407] Connection header received
0810 03:18:47.146190 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:47.146194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:47.146288 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:47.146378 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:18:47.146909 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:47.148043 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:47.150202 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:47.150741 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:47.150892 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:47.151581 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:18:47.151729 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:18:47.151865 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:47.152087 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 03:18:47.152199 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:18:47.165359 22642 raft_consensus.cc:1261] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Refusing update from remote peer 0e68e0fd34df4ab1bca59d36228cb248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:47.166097 22939 consensus_queue.cc:1035] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0b66e0fa6d845dab4da4a135f54115b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:18:47.171737 22494 raft_consensus.cc:1261] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Refusing update from remote peer 0e68e0fd34df4ab1bca59d36228cb248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:18:47.172538 22947 consensus_queue.cc:1035] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef888aa00e35409d964caf74d09d5eff" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:18:47.196354 22932 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 03:18:48.812424 22323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:48.812700 22323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:18:48.817992 22323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad11732858494c28abc6f6e5f1f01ed2: Sending DeleteTablet for 3 replicas of tablet 3aeb4caf9dd2470e996edf28ca5838a0
I20260810 03:18:48.819417 22473 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aeb4caf9dd2470e996edf28ca5838a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:48 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38868
I20260810 03:18:48.819532 22769 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aeb4caf9dd2470e996edf28ca5838a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:48 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:37326
I20260810 03:18:48.819572 22623 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aeb4caf9dd2470e996edf28ca5838a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:18:48 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260810 03:18:48.820801 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22401
I20260810 03:18:48.822357 23023 tablet_replica.cc:330] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: stopping tablet replica
I20260810 03:18:48.822656 23023 raft_consensus.cc:2229] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:48.823125 23023 raft_consensus.cc:2258] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:48.824980 23023 ts_tablet_manager.cc:1905] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:48.834750 23023 ts_tablet_manager.cc:1918] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:18:48.834964 23023 log.cc:1196] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/3aeb4caf9dd2470e996edf28ca5838a0
I20260810 03:18:48.835548 23023 ts_tablet_manager.cc:1939] T 3aeb4caf9dd2470e996edf28ca5838a0 P b0b66e0fa6d845dab4da4a135f54115b: Deleting consensus metadata
I20260810 03:18:48.837064 22309 catalog_manager.cc:4841] TS b0b66e0fa6d845dab4da4a135f54115b (127.19.246.194:41735): tablet 3aeb4caf9dd2470e996edf28ca5838a0 (table test-table [id=5e991cc7186d4b29a642758a0f425da4]) successfully deleted
I20260810 03:18:48.838148 23022 tablet_replica.cc:330] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: stopping tablet replica
I20260810 03:18:48.838388 23021 tablet_replica.cc:330] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: stopping tablet replica
I20260810 03:18:48.838373 23022 raft_consensus.cc:2229] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:18:48.838620 23021 raft_consensus.cc:2229] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:18:48.838770 23022 raft_consensus.cc:2258] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:48.838919 23021 raft_consensus.cc:2258] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:18:48.839968 23021 ts_tablet_manager.cc:1905] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:48.840332 23022 ts_tablet_manager.cc:1905] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:18:48.848731 23021 ts_tablet_manager.cc:1918] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:18:48.848941 23021 log.cc:1196] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/3aeb4caf9dd2470e996edf28ca5838a0
I20260810 03:18:48.849117 23022 ts_tablet_manager.cc:1918] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:18:48.849290 23022 log.cc:1196] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/3aeb4caf9dd2470e996edf28ca5838a0
I20260810 03:18:48.849572 23021 ts_tablet_manager.cc:1939] T 3aeb4caf9dd2470e996edf28ca5838a0 P ef888aa00e35409d964caf74d09d5eff: Deleting consensus metadata
I20260810 03:18:48.849817 23022 ts_tablet_manager.cc:1939] T 3aeb4caf9dd2470e996edf28ca5838a0 P 0e68e0fd34df4ab1bca59d36228cb248: Deleting consensus metadata
I20260810 03:18:48.851025 22310 catalog_manager.cc:4841] TS 0e68e0fd34df4ab1bca59d36228cb248 (127.19.246.195:36001): tablet 3aeb4caf9dd2470e996edf28ca5838a0 (table test-table [id=5e991cc7186d4b29a642758a0f425da4]) successfully deleted
I20260810 03:18:48.851034 22306 catalog_manager.cc:4841] TS ef888aa00e35409d964caf74d09d5eff (127.19.246.193:45521): tablet 3aeb4caf9dd2470e996edf28ca5838a0 (table test-table [id=5e991cc7186d4b29a642758a0f425da4]) successfully deleted
Aug 10 03:18:48 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:18:48.862414 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:48.822403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.19.246.193:45521)
0810 03:18:48.831388 (+ 8985us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:48.831398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:48.834885 (+ 3487us) server_negotiation.cc:407] Connection header received
0810 03:18:48.834979 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:48.834986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:48.835106 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:48.835228 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:48.837471 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:48.838822 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:48.841263 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:48.842226 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:48.842465 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:48.853621 (+ 11156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:48.853644 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:48.853661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:48.853715 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:48.858008 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:48.859772 (+ 1764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:48.859783 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:48.859786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:48.859974 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:48.860619 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:48.860624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:48.860626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:48.861070 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:48.861359 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:48.861753 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 03:18:48.861965 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8704,"thread_start_us":8234,"threads_started":1}
W20260810 03:18:48.863190 22540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56310
W20260810 03:18:48.864173 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22401: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:18:48.864454 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 22401
I20260810 03:18:48.893997 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22552
Aug 10 03:18:48 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:18:48.913812 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:48.894918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.19.246.194:41735)
0810 03:18:48.895574 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:48.895798 (+ 224us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:48.895821 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:18:48.895883 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:48.895887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:48.895965 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:48.896078 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:48.897680 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:48.898491 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:48.899997 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:48.900420 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:48.900604 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:48.906026 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:48.906049 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:48.906065 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:48.906118 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:48.909943 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:48.910886 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:48.910894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:48.910896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:48.911093 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:48.911796 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:48.911801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:48.911803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:48.912258 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:48.912555 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:48.913064 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 03:18:48.913309 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":342,"threads_started":1}
W20260810 03:18:48.914574 22689 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49892
W20260810 03:18:48.915477 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22552: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:18:48.915716 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 22552
I20260810 03:18:48.945449 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22699
Aug 10 03:18:48 dist-test-slave-btw8 krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:18:48.966625 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:48.946463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.19.246.195:36001)
0810 03:18:48.947225 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:48.947230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:48.947247 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:18:48.947302 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:48.947306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:48.947437 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:48.947535 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:48.949444 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:48.950228 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:48.952397 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:48.953061 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:48.953460 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:48.959812 (+ 6352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:48.959834 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:48.959853 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:48.959905 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:48.963221 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:48.964233 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:48.964241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:48.964243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:48.964397 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:48.964875 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:48.964882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:48.964885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:48.965296 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:48.965680 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:48.965998 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:18:48.966262 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":335,"threads_started":1}
W20260810 03:18:48.967522 22837 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37914
W20260810 03:18:48.968325 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22699: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:18:48.968544 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 22699
I20260810 03:18:49.000900 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 22290
I20260810 03:18:49.002349 22369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:49.141883 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 22290
2026-08-10T03:18:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9095 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:18:49 dist-test-slave-btw8 krb5kdc[23032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:18:49 dist-test-slave-btw8 krb5kdc[23032](info): set up 2 sockets
Aug 10 03:18:49 dist-test-slave-btw8 krb5kdc[23032](info): commencing operation
krb5kdc: starting...
W20260810 03:18:50.478749 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260810 03:18:50.681807 20443 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.203s user 0.000s sys 0.008s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260810 03:18:50.786046 20443 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.104s user 0.000s sys 0.008s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 03:18:50.901751 20443 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.115s user 0.000s sys 0.008s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/test-user.keytab.
W20260810 03:18:51.543000 20443 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.641s user 0.000s sys 0.008s
Aug 10 03:18:51 dist-test-slave-btw8 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:18:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:18:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:18:51.698091 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:44823
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:44823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:52.174083 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:52.174433 23047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:52.174584 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:52.183710 23047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:18:52.183861 23047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:52.183967 23047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:52.184069 23047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:18:52.184258 23047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:18:52.196568 23047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:44823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:44823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:52.199052 23047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:52.201489 23047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:18:52.225308 23047 server_base.cc:1034] running on GCE node
W20260810 03:18:52.225380 23054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:52.225611 23053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:52.225382 23056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:52.226910 23047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:52.228832 23047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:52.230422 23047 hybrid_clock.cc:648] HybridClock initialized: now 1786331932230307 us; error 113 us; skew 500 ppm
I20260810 03:18:52.231019 23047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:52 dist-test-slave-btw8 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:52.235864 23047 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:18:52.238790 23047 webserver.cc:460] Webserver started at http://127.19.246.254:33217/ using document root <none> and password file <none>
I20260810 03:18:52.239907 23047 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:52.240056 23047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:52.240502 23047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:52.245198 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "194f9a85fffa49989a898dc01bafa584"
format_stamp: "Formatted at 2026-08-10 03:18:52 on dist-test-slave-btw8"
server_key: "f684347c11979de32962968fd173bc15"
server_key_iv: "4dcbd3a9ec9b512e87fd0ac6d8442122"
server_key_version: "encryptionkey@0"
I20260810 03:18:52.246439 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "194f9a85fffa49989a898dc01bafa584"
format_stamp: "Formatted at 2026-08-10 03:18:52 on dist-test-slave-btw8"
server_key: "f684347c11979de32962968fd173bc15"
server_key_iv: "4dcbd3a9ec9b512e87fd0ac6d8442122"
server_key_version: "encryptionkey@0"
I20260810 03:18:52.253765 23047 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260810 03:18:52.259892 23063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:52.262058 23047 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:18:52.262359 23047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "194f9a85fffa49989a898dc01bafa584"
format_stamp: "Formatted at 2026-08-10 03:18:52 on dist-test-slave-btw8"
server_key: "f684347c11979de32962968fd173bc15"
server_key_iv: "4dcbd3a9ec9b512e87fd0ac6d8442122"
server_key_version: "encryptionkey@0"
I20260810 03:18:52.262656 23047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:52.280812 23047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:52.287827 23047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:52.288268 23047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:52.313970 23047 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:44823
I20260810 03:18:52.314098 23118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:44823 every 8 connection(s)
I20260810 03:18:52.316625 23047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:18:52.321925 23121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:52.326268 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 23047
I20260810 03:18:52.326560 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:18:52.327493 20443 external_mini_cluster.cc:1428] Setting key dcae1e563bbdb7c90348bca5fb59963f
I20260810 03:18:52.336292 23121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584: Bootstrap starting.
I20260810 03:18:52.340787 23121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:52.342768 23121 log.cc:824] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:52.348611 23121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584: No bootstrap required, opened a new log
Aug 10 03:18:52 dist-test-slave-btw8 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:52.357913 23121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194f9a85fffa49989a898dc01bafa584" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44823 } }
I20260810 03:18:52.358479 23121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:52.358625 23121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194f9a85fffa49989a898dc01bafa584, State: Initialized, Role: FOLLOWER
I20260810 03:18:52.359341 23121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194f9a85fffa49989a898dc01bafa584" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44823 } }
I20260810 03:18:52.359701 23121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:18:52.359855 23121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:18:52.360085 23121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:52.362627 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:52.330282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.19.246.254:44823)
0810 03:18:52.331310 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:52.331324 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:52.331452 (+ 128us) server_negotiation.cc:407] Connection header received
0810 03:18:52.335221 (+ 3769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:52.335256 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:52.336806 (+ 1550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:52.337338 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:52.339580 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:52.341221 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:52.344541 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:52.346025 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:52.346320 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:52.352265 (+ 5945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:52.352276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:52.352299 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:52.352351 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:52.357027 (+ 4676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:52.358138 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:52.358147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:52.358155 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:52.358403 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:52.358988 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:52.358993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:52.358998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:52.359982 (+ 984us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:52.360347 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:52.360795 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 03:18:52.361341 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":488,"threads_started":1}
I20260810 03:18:52.363090 23121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194f9a85fffa49989a898dc01bafa584" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44823 } }
I20260810 03:18:52.363754 23121 leader_election.cc:302] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 194f9a85fffa49989a898dc01bafa584; no voters:
I20260810 03:18:52.364470 23121 leader_election.cc:288] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:18:52.364703 23127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:52.366680 23127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [term 1 LEADER]: Becoming Leader. State: Replica: 194f9a85fffa49989a898dc01bafa584, State: Running, Role: LEADER
I20260810 03:18:52.366987 23121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:18:52.367476 23127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194f9a85fffa49989a898dc01bafa584" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44823 } }
I20260810 03:18:52.371402 23128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "194f9a85fffa49989a898dc01bafa584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194f9a85fffa49989a898dc01bafa584" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44823 } } }
I20260810 03:18:52.371975 23128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:52.371480 23129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 194f9a85fffa49989a898dc01bafa584. Latest consensus state: current_term: 1 leader_uuid: "194f9a85fffa49989a898dc01bafa584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194f9a85fffa49989a898dc01bafa584" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44823 } } }
I20260810 03:18:52.372360 23129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584 [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:52.376780 23135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:18:52.382958 23135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:18:52.395800 23135 catalog_manager.cc:1347] Generated new cluster ID: 1482dc9fba744b01a086a7f171855692
I20260810 03:18:52.395978 23135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:18:52.427591 23135 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:18:52.429450 23135 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:18:52.438036 23135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 194f9a85fffa49989a898dc01bafa584: Generated new TSK 0
I20260810 03:18:52.439177 23135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:18:52.527768 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:44823
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:52.921523 23148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:52.921844 23148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:52.921947 23148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:52.929612 23148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:52.929755 23148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:52.929872 23148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:18:52.941411 23148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:44823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:52.942955 23148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:52.944936 23148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:52.962085 23153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:52.962186 23154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:52.962136 23156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:52.965816 23148 server_base.cc:1034] running on GCE node
I20260810 03:18:52.966684 23148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:52.968096 23148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:52.969568 23148 hybrid_clock.cc:648] HybridClock initialized: now 1786331932969474 us; error 111 us; skew 500 ppm
I20260810 03:18:52.969997 23148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:52 dist-test-slave-btw8 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:52.974103 23148 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:18:52.976276 23148 webserver.cc:460] Webserver started at http://127.19.246.193:37925/ using document root <none> and password file <none>
I20260810 03:18:52.977398 23148 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:52.977641 23148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:52.978041 23148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:52.981698 23148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "0d1e9fe09ff04082ae63ddb94fb91158"
format_stamp: "Formatted at 2026-08-10 03:18:52 on dist-test-slave-btw8"
server_key: "cf6c42396a877ea8655eb8719a2187b7"
server_key_iv: "876d197ddd302d46069228931d6b4127"
server_key_version: "encryptionkey@0"
I20260810 03:18:52.982800 23148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "0d1e9fe09ff04082ae63ddb94fb91158"
format_stamp: "Formatted at 2026-08-10 03:18:52 on dist-test-slave-btw8"
server_key: "cf6c42396a877ea8655eb8719a2187b7"
server_key_iv: "876d197ddd302d46069228931d6b4127"
server_key_version: "encryptionkey@0"
I20260810 03:18:52.989825 23148 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260810 03:18:52.994319 23163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:52.996156 23148 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:18:52.996529 23148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "0d1e9fe09ff04082ae63ddb94fb91158"
format_stamp: "Formatted at 2026-08-10 03:18:52 on dist-test-slave-btw8"
server_key: "cf6c42396a877ea8655eb8719a2187b7"
server_key_iv: "876d197ddd302d46069228931d6b4127"
server_key_version: "encryptionkey@0"
I20260810 03:18:52.996886 23148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:53.030257 23148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:53.035669 23148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:53.036029 23148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:53.037225 23148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:53.038883 23148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:53.039007 23148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:53.039187 23148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:53.039283 23148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:53.079178 23148 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:40391
I20260810 03:18:53.079306 23286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:40391 every 8 connection(s)
I20260810 03:18:53.080708 23148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:18:53.087831 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 23148
I20260810 03:18:53.088089 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:18:53.088784 20443 external_mini_cluster.cc:1428] Setting key e546681340ad54824f74925bb00bad9d
Aug 10 03:18:53 dist-test-slave-btw8 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:53.106159 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:53.083892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:53807 (local address 127.19.246.254:44823)
0810 03:18:53.084475 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:53.084481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:53.085633 (+ 1152us) server_negotiation.cc:407] Connection header received
0810 03:18:53.087964 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:53.087971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:53.088102 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:53.088252 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:53.090346 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:53.091259 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:53.093165 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:53.093632 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:53.093795 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:53.098267 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:53.098275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:53.098282 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:53.098337 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:53.102171 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:53.103216 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:53.103223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:53.103227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:53.103451 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:53.103974 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:53.103983 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:53.103987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:53.104323 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:53.104507 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:53.105414 (+ 907us) server_negotiation.cc:299] Negotiation successful
0810 03:18:53.105707 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":475,"thread_start_us":336,"threads_started":1}
I20260810 03:18:53.107432 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:53.084534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:44823 (local address 127.19.246.193:53807)
0810 03:18:53.085432 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:53.085477 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:53.087623 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:53.088573 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:53.088583 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:53.089294 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:53.090146 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:53.090171 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:53.091488 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:53.091496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:53.092958 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:53.092979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:53.093927 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:53.093936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:53.094225 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:53.095053 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:53.095080 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:53.097955 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:53.102358 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:53.102377 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:53.102395 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:53.103049 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:53.103591 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:53.103612 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:53.103615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:53.103836 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:53.104482 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:53.104492 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:53.104966 (+ 474us) client_negotiation.cc:769] Sending connection context
0810 03:18:53.105311 (+ 345us) client_negotiation.cc:240] Negotiation successful
0810 03:18:53.105980 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":330,"threads_started":1}
I20260810 03:18:53.109776 23287 heartbeater.cc:347] Connected to a master server at 127.19.246.254:44823
I20260810 03:18:53.110206 23287 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:53.111315 23287 heartbeater.cc:511] Master 127.19.246.254:44823 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
I20260810 03:18:53.114243 23081 ts_manager.cc:194] Registered new tserver with Master: 0d1e9fe09ff04082ae63ddb94fb91158 (127.19.246.193:40391)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
I20260810 03:18:53.116305 23081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:53807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:18:53.149411 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:44823
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:53.524900 23293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:53.525215 23293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:53.525312 23293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:53.532663 23293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:53.532802 23293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:53.532919 23293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:18:53.543088 23293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:44823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:53.544904 23293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:53.546236 23293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:53.561311 23299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:53.561475 23298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:53.561440 23301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:53.562991 23293 server_base.cc:1034] running on GCE node
I20260810 03:18:53.563855 23293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:53.564985 23293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:53.566213 23293 hybrid_clock.cc:648] HybridClock initialized: now 1786331933566169 us; error 40 us; skew 500 ppm
I20260810 03:18:53.566555 23293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:53 dist-test-slave-btw8 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331933, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:53.570572 23293 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:18:53.572535 23293 webserver.cc:460] Webserver started at http://127.19.246.194:36177/ using document root <none> and password file <none>
I20260810 03:18:53.573292 23293 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:53.573407 23293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:53.573688 23293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:53.577111 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "89b5f56aed1844908904feb4eeef124f"
format_stamp: "Formatted at 2026-08-10 03:18:53 on dist-test-slave-btw8"
server_key: "2eaf380f6d16d347d8637b7c3516dfc0"
server_key_iv: "cd700e15bfae3e4e5f4fe61cec7d3b7b"
server_key_version: "encryptionkey@0"
I20260810 03:18:53.578120 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "89b5f56aed1844908904feb4eeef124f"
format_stamp: "Formatted at 2026-08-10 03:18:53 on dist-test-slave-btw8"
server_key: "2eaf380f6d16d347d8637b7c3516dfc0"
server_key_iv: "cd700e15bfae3e4e5f4fe61cec7d3b7b"
server_key_version: "encryptionkey@0"
I20260810 03:18:53.584371 23293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:18:53.588894 23308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:53.590544 23293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260810 03:18:53.590801 23293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "89b5f56aed1844908904feb4eeef124f"
format_stamp: "Formatted at 2026-08-10 03:18:53 on dist-test-slave-btw8"
server_key: "2eaf380f6d16d347d8637b7c3516dfc0"
server_key_iv: "cd700e15bfae3e4e5f4fe61cec7d3b7b"
server_key_version: "encryptionkey@0"
I20260810 03:18:53.591018 23293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:53.609934 23293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:53.614993 23293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:53.615290 23293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:53.616551 23293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:53.618239 23293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:53.618364 23293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:53.618534 23293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:53.618613 23293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:53.657430 23293 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:46221
I20260810 03:18:53.657482 23427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:46221 every 8 connection(s)
I20260810 03:18:53.658798 23293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:18:53.667076 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 23293
I20260810 03:18:53.667316 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:18:53.667827 20443 external_mini_cluster.cc:1428] Setting key 04851225473cf96df24951561f3cf5ea
Aug 10 03:18:53 dist-test-slave-btw8 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331933, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:53.682169 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:53.661736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:37085 (local address 127.19.246.254:44823)
0810 03:18:53.662251 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:53.662255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:53.663005 (+ 750us) server_negotiation.cc:407] Connection header received
0810 03:18:53.664992 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:53.664998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:53.665099 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:53.665225 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:53.666973 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:53.668146 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:53.670002 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:53.670436 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:53.670570 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:53.674788 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:53.674793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:53.674796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:53.674833 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:53.678563 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:53.679695 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:53.679702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:53.679705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:53.679850 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:53.680326 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:53.680331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:53.680335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:53.680637 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:53.680799 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:53.681539 (+ 740us) server_negotiation.cc:299] Negotiation successful
0810 03:18:53.681775 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":267,"threads_started":1}
I20260810 03:18:53.683210 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:53.662051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:44823 (local address 127.19.246.194:37085)
0810 03:18:53.662845 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:53.662884 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:53.664701 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:53.665486 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:53.665495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:53.666073 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:53.666731 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:53.666759 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:53.668431 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:53.668437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:53.669840 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:53.669852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:53.670685 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:53.670693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:53.670930 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:53.671731 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:53.671754 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:53.674471 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:53.678794 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:53.678805 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:53.678826 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:53.679548 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:53.679974 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:53.679979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:53.679982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:53.680196 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:53.680774 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:53.680784 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:53.681169 (+ 385us) client_negotiation.cc:769] Sending connection context
0810 03:18:53.681450 (+ 281us) client_negotiation.cc:240] Negotiation successful
0810 03:18:53.682020 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":283,"threads_started":1}
I20260810 03:18:53.684942 23428 heartbeater.cc:347] Connected to a master server at 127.19.246.254:44823
I20260810 03:18:53.685312 23428 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:53.686115 23428 heartbeater.cc:511] Master 127.19.246.254:44823 requested a full tablet report, sending...
I20260810 03:18:53.687830 23081 ts_manager.cc:194] Registered new tserver with Master: 89b5f56aed1844908904feb4eeef124f (127.19.246.194:46221)
I20260810 03:18:53.689378 23081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:37085
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:18:53.728329 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:44823
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:18:54.120795 23287 heartbeater.cc:503] Master 127.19.246.254:44823 was elected leader, sending a full tablet report...
W20260810 03:18:54.131047 23435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:54.131505 23435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:54.131680 23435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:54.140559 23435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:54.140748 23435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:54.140942 23435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:18:54.151723 23435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:44823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:54.153244 23435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:54.154816 23435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:54.174631 23442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:54.174757 23444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:54.174758 23441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:54.175031 23435 server_base.cc:1034] running on GCE node
I20260810 03:18:54.176057 23435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:54.177321 23435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:54.178644 23435 hybrid_clock.cc:648] HybridClock initialized: now 1786331934178524 us; error 60 us; skew 500 ppm
I20260810 03:18:54.179118 23435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:54 dist-test-slave-btw8 krb5kdc[23032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331934, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:54.189471 23435 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:18:54.191973 23435 webserver.cc:460] Webserver started at http://127.19.246.195:41109/ using document root <none> and password file <none>
I20260810 03:18:54.192798 23435 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:54.192919 23435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:54.193203 23435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:54.197165 23435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "bd089d78b3c842f4ab0a0c4e6de09b48"
format_stamp: "Formatted at 2026-08-10 03:18:54 on dist-test-slave-btw8"
server_key: "763d7ba4ffcce848c29af2a47aa86db3"
server_key_iv: "e2743112e7405e6f426851e26d12bfa1"
server_key_version: "encryptionkey@0"
I20260810 03:18:54.198331 23435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "bd089d78b3c842f4ab0a0c4e6de09b48"
format_stamp: "Formatted at 2026-08-10 03:18:54 on dist-test-slave-btw8"
server_key: "763d7ba4ffcce848c29af2a47aa86db3"
server_key_iv: "e2743112e7405e6f426851e26d12bfa1"
server_key_version: "encryptionkey@0"
I20260810 03:18:54.206838 23435 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260810 03:18:54.212244 23451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:54.214056 23435 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:18:54.214241 23435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "bd089d78b3c842f4ab0a0c4e6de09b48"
format_stamp: "Formatted at 2026-08-10 03:18:54 on dist-test-slave-btw8"
server_key: "763d7ba4ffcce848c29af2a47aa86db3"
server_key_iv: "e2743112e7405e6f426851e26d12bfa1"
server_key_version: "encryptionkey@0"
I20260810 03:18:54.214543 23435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:54.240159 23435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:54.245837 23435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:54.246209 23435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:54.247515 23435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:54.249300 23435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:54.249393 23435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:54.249536 23435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:54.249596 23435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:54.289652 23435 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:37633
I20260810 03:18:54.289758 23574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:37633 every 8 connection(s)
I20260810 03:18:54.291169 23435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:18:54.301083 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 23435
I20260810 03:18:54.301316 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:18:54.302278 20443 external_mini_cluster.cc:1428] Setting key 5c17518ed5e6c262e8b0d88e50824799
Aug 10 03:18:54 dist-test-slave-btw8 krb5kdc[23032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331934, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:54.320451 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:54.295934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:40363 (local address 127.19.246.254:44823)
0810 03:18:54.296545 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:54.296550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:54.298132 (+ 1582us) server_negotiation.cc:407] Connection header received
0810 03:18:54.300367 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:54.300373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:54.300483 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:54.300617 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:54.303942 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:54.305123 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:54.307306 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:54.307928 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:54.308047 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:54.312424 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:54.312430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:54.312434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:54.312476 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:54.315987 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:54.317203 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:54.317209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:54.317212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:54.317422 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:54.318059 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:54.318064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:54.318068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:54.318459 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:54.318621 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:54.319724 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0810 03:18:54.319981 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":299,"threads_started":1}
I20260810 03:18:54.321609 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:54.296722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:44823 (local address 127.19.246.195:40363)
0810 03:18:54.297923 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:54.297974 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:54.300032 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:54.301062 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:54.301072 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:54.302642 (+ 1570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:54.303515 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:54.303558 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:54.305320 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:54.305326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:54.307114 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:54.307138 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:54.308216 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:54.308222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:54.308469 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:54.309295 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:54.309320 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:54.312118 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:54.316195 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:54.316208 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:54.316230 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:54.316989 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:54.317560 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:54.317568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:54.317571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:54.317908 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:54.318588 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:54.318598 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:54.319183 (+ 585us) client_negotiation.cc:769] Sending connection context
0810 03:18:54.319626 (+ 443us) client_negotiation.cc:240] Negotiation successful
0810 03:18:54.320244 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":581,"threads_started":1}
I20260810 03:18:54.323333 23575 heartbeater.cc:347] Connected to a master server at 127.19.246.254:44823
I20260810 03:18:54.323877 23575 heartbeater.cc:464] Registering TS with master...
I20260810 03:18:54.324630 23575 heartbeater.cc:511] Master 127.19.246.254:44823 requested a full tablet report, sending...
I20260810 03:18:54.326359 23081 ts_manager.cc:194] Registered new tserver with Master: bd089d78b3c842f4ab0a0c4e6de09b48 (127.19.246.195:37633)
I20260810 03:18:54.327260 23081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:40363
I20260810 03:18:54.332417 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:18:54.375649 23577 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:18:54.372615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.19.246.254:44823)
0810 03:18:54.372813 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:54.372818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:54.373390 (+ 572us) server_negotiation.cc:407] Connection header received
0810 03:18:54.373623 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:54.373628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:54.373720 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:54.373832 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:54.375311 (+ 1479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36814: BlockingRecv error: recv got EOF from 127.0.0.1:36814 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:18:54.379156 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23148
W20260810 03:18:54.382480 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23148: Not authorized: Client connection negotiation failed: client connection to 127.19.246.193:40391: server requires authentication, but client does not have Kerberos credentials available
I20260810 03:18:54.382736 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 23148
I20260810 03:18:54.403645 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23293
W20260810 03:18:54.407008 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23293: Not authorized: Client connection negotiation failed: client connection to 127.19.246.194:46221: server requires authentication, but client does not have Kerberos credentials available
I20260810 03:18:54.407238 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 23293
I20260810 03:18:54.426155 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23435
W20260810 03:18:54.429695 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23435: Not authorized: Client connection negotiation failed: client connection to 127.19.246.195:37633: server requires authentication, but client does not have Kerberos credentials available
I20260810 03:18:54.429874 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 23435
I20260810 03:18:54.448942 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23047
I20260810 03:18:54.450352 23114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:54.560714 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 23047
2026-08-10T03:18:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5411 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:18:54 dist-test-slave-btw8 krb5kdc[23597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:18:54 dist-test-slave-btw8 krb5kdc[23597](info): set up 2 sockets
Aug 10 03:18:54 dist-test-slave-btw8 krb5kdc[23597](info): commencing operation
krb5kdc: starting...
W20260810 03:18:55.796065 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:18:55 dist-test-slave-btw8 krb5kdc[23597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:18:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:18:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:18:56.069480 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:35835
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:34591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:56.435247 23610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:56.435674 23610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:56.435773 23610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:56.443460 23610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:18:56.443655 23610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:56.443707 23610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:56.443760 23610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:18:56.443812 23610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:18:56.456326 23610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:35835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:56.457811 23610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:56.459506 23610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:56.473304 23618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:56.473316 23615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:56.473343 23616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:56.474262 23610 server_base.cc:1034] running on GCE node
I20260810 03:18:56.475097 23610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:56.476779 23610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:56.478015 23610 hybrid_clock.cc:648] HybridClock initialized: now 1786331936477967 us; error 45 us; skew 500 ppm
I20260810 03:18:56.478430 23610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:56 dist-test-slave-btw8 krb5kdc[23597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331936, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:56.482103 23610 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:18:56.483969 23610 webserver.cc:460] Webserver started at http://127.19.246.254:35869/ using document root <none> and password file <none>
I20260810 03:18:56.484731 23610 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:56.484889 23610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:56.485225 23610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:56.488618 23610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "569547b589a14b7eb8cdca6dfcf2690f"
format_stamp: "Formatted at 2026-08-10 03:18:56 on dist-test-slave-btw8"
server_key: "9c77244a450d88dc41aa377348449dbf"
server_key_iv: "c252a3323320c9777467b415e04afac5"
server_key_version: "encryptionkey@0"
I20260810 03:18:56.489543 23610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "569547b589a14b7eb8cdca6dfcf2690f"
format_stamp: "Formatted at 2026-08-10 03:18:56 on dist-test-slave-btw8"
server_key: "9c77244a450d88dc41aa377348449dbf"
server_key_iv: "c252a3323320c9777467b415e04afac5"
server_key_version: "encryptionkey@0"
I20260810 03:18:56.495702 23610 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 03:18:56.500363 23625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:56.502374 23610 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:18:56.502645 23610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "569547b589a14b7eb8cdca6dfcf2690f"
format_stamp: "Formatted at 2026-08-10 03:18:56 on dist-test-slave-btw8"
server_key: "9c77244a450d88dc41aa377348449dbf"
server_key_iv: "c252a3323320c9777467b415e04afac5"
server_key_version: "encryptionkey@0"
I20260810 03:18:56.502910 23610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:56.526221 23610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:56.531126 23610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:56.531509 23610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:56.552345 23610 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:35835
I20260810 03:18:56.552388 23678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:35835 every 8 connection(s)
I20260810 03:18:56.553869 23610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:18:56.556829 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 23610
I20260810 03:18:56.557119 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:18:56.557782 20443 external_mini_cluster.cc:1428] Setting key b65d0e606f27a2f66b801d59626eb795
I20260810 03:18:56.558209 23679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:56.574334 23679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f: Bootstrap starting.
Aug 10 03:18:56 dist-test-slave-btw8 krb5kdc[23597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:56.577803 23679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:56.578935 23679 log.cc:824] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:18:56.582010 23679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f: No bootstrap required, opened a new log
I20260810 03:18:56.583179 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:56.559941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.19.246.254:35835)
0810 03:18:56.561180 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:56.561193 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:56.561242 (+ 49us) server_negotiation.cc:407] Connection header received
0810 03:18:56.563223 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:56.563247 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:56.563765 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:56.564131 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:56.566170 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:56.567619 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:56.569571 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:56.570330 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:56.570523 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:56.575604 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:56.575618 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:56.575637 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:56.575683 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:56.578945 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:56.579788 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:56.579794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:56.579800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:56.579975 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:56.580421 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:56.580424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:56.580427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:56.580954 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:56.581295 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:56.581709 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 03:18:56.582175 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":460,"threads_started":1}
I20260810 03:18:56.587311 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:18:56.590349 23679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569547b589a14b7eb8cdca6dfcf2690f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35835 } }
I20260810 03:18:56.590775 23679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:56.590904 23679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569547b589a14b7eb8cdca6dfcf2690f, State: Initialized, Role: FOLLOWER
I20260810 03:18:56.591655 23679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569547b589a14b7eb8cdca6dfcf2690f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35835 } }
I20260810 03:18:56.591928 23679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:18:56.592063 23679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:18:56.592240 23679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:56.594664 23679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569547b589a14b7eb8cdca6dfcf2690f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35835 } }
I20260810 03:18:56.595239 23679 leader_election.cc:302] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 569547b589a14b7eb8cdca6dfcf2690f; no voters:
I20260810 03:18:56.595989 23679 leader_election.cc:288] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:18:56.596084 23686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:56.597774 23686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [term 1 LEADER]: Becoming Leader. State: Replica: 569547b589a14b7eb8cdca6dfcf2690f, State: Running, Role: LEADER
I20260810 03:18:56.598855 23686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569547b589a14b7eb8cdca6dfcf2690f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35835 } }
I20260810 03:18:56.599618 23679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:18:56.602979 23688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "569547b589a14b7eb8cdca6dfcf2690f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569547b589a14b7eb8cdca6dfcf2690f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35835 } } }
I20260810 03:18:56.602963 23689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569547b589a14b7eb8cdca6dfcf2690f. Latest consensus state: current_term: 1 leader_uuid: "569547b589a14b7eb8cdca6dfcf2690f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569547b589a14b7eb8cdca6dfcf2690f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35835 } } }
I20260810 03:18:56.603500 23689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:56.603497 23688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:56.616685 23693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:18:56.622881 23693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:18:56.648146 23693 catalog_manager.cc:1347] Generated new cluster ID: 48bf7433013c4944ae676edc42b67e43
I20260810 03:18:56.648342 23693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:18:56.662963 23712 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:18:56.674065 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:56.663654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.19.246.254:35835)
0810 03:18:56.663827 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:56.663832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:56.664593 (+ 761us) server_negotiation.cc:407] Connection header received
0810 03:18:56.664748 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:56.664755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:56.664847 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:56.664982 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:56.666225 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:56.667441 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:56.669347 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:56.670117 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:56.670282 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:56.671618 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:56.671625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:56.671639 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:18:56.672943 (+ 1304us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:56.673017 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:18:56.673059 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:56.673193 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:56.673326 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 03:18:56.673452 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:18:56.682256 23693 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:18:56.683879 23693 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:18:56.700002 23693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 569547b589a14b7eb8cdca6dfcf2690f: Generated new TSK 0
I20260810 03:18:56.701053 23693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:18:56.716252 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23610
I20260810 03:18:56.717653 23674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:18:56.828994 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 23610
2026-08-10T03:18:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2258 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:18:56 dist-test-slave-btw8 krb5kdc[23718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:18:56 dist-test-slave-btw8 krb5kdc[23718](info): set up 2 sockets
Aug 10 03:18:56 dist-test-slave-btw8 krb5kdc[23718](info): commencing operation
krb5kdc: starting...
W20260810 03:18:58.022897 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:18:58 dist-test-slave-btw8 krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:18:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:18:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:18:58.297529 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:43709
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:43709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:58.672770 23731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:58.673164 23731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:58.673319 23731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:58.681650 23731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:18:58.681867 23731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:58.682044 23731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:58.682159 23731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:18:58.682243 23731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:18:58.693557 23731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:43709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:58.695128 23731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:58.696628 23731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:58.711794 23740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:58.711759 23737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:58.711831 23736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:58.712917 23731 server_base.cc:1034] running on GCE node
I20260810 03:18:58.713922 23731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:58.715859 23731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:58.717151 23731 hybrid_clock.cc:648] HybridClock initialized: now 1786331938717108 us; error 52 us; skew 500 ppm
I20260810 03:18:58.717586 23731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:58 dist-test-slave-btw8 krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331938, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:58.721197 23731 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:18:58.723016 23731 webserver.cc:460] Webserver started at http://127.19.246.254:34645/ using document root <none> and password file <none>
I20260810 03:18:58.723856 23731 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:58.724020 23731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:58.724421 23731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:58.727895 23731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "3ae569b9a76445a699fd2df8c5516336"
format_stamp: "Formatted at 2026-08-10 03:18:58 on dist-test-slave-btw8"
server_key: "54b2dc98a323423917f785a82fb6fe7e"
server_key_iv: "2c80d6d4fbc72b59b817900b0c987065"
server_key_version: "encryptionkey@0"
I20260810 03:18:58.728787 23731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "3ae569b9a76445a699fd2df8c5516336"
format_stamp: "Formatted at 2026-08-10 03:18:58 on dist-test-slave-btw8"
server_key: "54b2dc98a323423917f785a82fb6fe7e"
server_key_iv: "2c80d6d4fbc72b59b817900b0c987065"
server_key_version: "encryptionkey@0"
I20260810 03:18:58.735277 23731 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:18:58.739892 23747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:58.742076 23731 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:18:58.742352 23731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "3ae569b9a76445a699fd2df8c5516336"
format_stamp: "Formatted at 2026-08-10 03:18:58 on dist-test-slave-btw8"
server_key: "54b2dc98a323423917f785a82fb6fe7e"
server_key_iv: "2c80d6d4fbc72b59b817900b0c987065"
server_key_version: "encryptionkey@0"
I20260810 03:18:58.742640 23731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:58.767295 23731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:58.772646 23731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:58.773026 23731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:58.796267 23731 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:43709
I20260810 03:18:58.796305 23804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:43709 every 8 connection(s)
I20260810 03:18:58.797827 23731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:18:58.802042 23805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:18:58.805644 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 23731
I20260810 03:18:58.805871 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:18:58.806572 20443 external_mini_cluster.cc:1428] Setting key 7e98f6b2890968133dddaf82059cd454
I20260810 03:18:58.816633 23805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336: Bootstrap starting.
I20260810 03:18:58.820238 23805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336: Neither blocks nor log segments found. Creating new log.
I20260810 03:18:58.821396 23805 log.cc:824] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:18:58 dist-test-slave-btw8 krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:58.825063 23805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336: No bootstrap required, opened a new log
I20260810 03:18:58.835421 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:58.808921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.19.246.254:43709)
0810 03:18:58.809807 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:58.809820 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:58.809865 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:18:58.811929 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:58.811959 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:58.812478 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:58.813210 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:58.815175 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:58.816875 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:58.818422 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:58.819288 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:58.819597 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:58.824904 (+ 5307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:58.824917 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:58.824939 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:58.824988 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:58.829773 (+ 4785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:58.830737 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:58.830753 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:58.830763 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:58.831019 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:58.831568 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:58.831574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:58.831577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:58.832424 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:58.832756 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:58.833257 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 03:18:58.833761 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":383,"threads_started":1}
I20260810 03:18:58.836608 23805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae569b9a76445a699fd2df8c5516336" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43709 } }
I20260810 03:18:58.837131 23805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:18:58.837255 23805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae569b9a76445a699fd2df8c5516336, State: Initialized, Role: FOLLOWER
I20260810 03:18:58.837905 23805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae569b9a76445a699fd2df8c5516336" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43709 } }
I20260810 03:18:58.838161 23805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:18:58.838315 23805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:18:58.838510 23805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:18:58.841518 23805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae569b9a76445a699fd2df8c5516336" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43709 } }
I20260810 03:18:58.842216 23805 leader_election.cc:302] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ae569b9a76445a699fd2df8c5516336; no voters:
I20260810 03:18:58.842953 23805 leader_election.cc:288] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:18:58.843169 23811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:18:58.844099 23811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [term 1 LEADER]: Becoming Leader. State: Replica: 3ae569b9a76445a699fd2df8c5516336, State: Running, Role: LEADER
I20260810 03:18:58.844820 23811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae569b9a76445a699fd2df8c5516336" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43709 } }
I20260810 03:18:58.845731 23805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:18:58.848742 23812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ae569b9a76445a699fd2df8c5516336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae569b9a76445a699fd2df8c5516336" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43709 } } }
I20260810 03:18:58.848832 23813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ae569b9a76445a699fd2df8c5516336. Latest consensus state: current_term: 1 leader_uuid: "3ae569b9a76445a699fd2df8c5516336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae569b9a76445a699fd2df8c5516336" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43709 } } }
I20260810 03:18:58.851872 23812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:58.851888 23813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336 [sys.catalog]: This master's current role is: LEADER
I20260810 03:18:58.857051 23822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:18:58.864522 23822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:18:58.875308 23822 catalog_manager.cc:1347] Generated new cluster ID: 454b4072ea5b4a6f91c2aa37decd033c
I20260810 03:18:58.875519 23822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:18:58.888288 23822 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:18:58.890398 23822 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:18:58.902451 23822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ae569b9a76445a699fd2df8c5516336: Generated new TSK 0
I20260810 03:18:58.903770 23822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:18:58.975801 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43709
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:59.352044 23837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:59.352366 23837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:59.352512 23837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:59.360893 23837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:59.361027 23837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:59.361225 23837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:18:59.372119 23837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:59.373548 23837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:59.375000 23837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:59.390630 23844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:59.390966 23842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:59.391165 23846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:59.392884 23837 server_base.cc:1034] running on GCE node
I20260810 03:18:59.393972 23837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:59.395185 23837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:59.396448 23837 hybrid_clock.cc:648] HybridClock initialized: now 1786331939396371 us; error 76 us; skew 500 ppm
I20260810 03:18:59.396873 23837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:59 dist-test-slave-btw8 krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331939, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:59.410871 23837 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:18:59.412919 23837 webserver.cc:460] Webserver started at http://127.19.246.193:37351/ using document root <none> and password file <none>
I20260810 03:18:59.413780 23837 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:59.413964 23837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:59.414348 23837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:59.417969 23837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "64f8d661edc54dc6aa9be591e15ae4ce"
format_stamp: "Formatted at 2026-08-10 03:18:59 on dist-test-slave-btw8"
server_key: "269b0e64e428b48bc75b3f6caf18f75a"
server_key_iv: "218919fc6db9ff72898686d237a6de60"
server_key_version: "encryptionkey@0"
I20260810 03:18:59.418941 23837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "64f8d661edc54dc6aa9be591e15ae4ce"
format_stamp: "Formatted at 2026-08-10 03:18:59 on dist-test-slave-btw8"
server_key: "269b0e64e428b48bc75b3f6caf18f75a"
server_key_iv: "218919fc6db9ff72898686d237a6de60"
server_key_version: "encryptionkey@0"
I20260810 03:18:59.425091 23837 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20260810 03:18:59.429412 23854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:59.431037 23837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:18:59.431283 23837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "64f8d661edc54dc6aa9be591e15ae4ce"
format_stamp: "Formatted at 2026-08-10 03:18:59 on dist-test-slave-btw8"
server_key: "269b0e64e428b48bc75b3f6caf18f75a"
server_key_iv: "218919fc6db9ff72898686d237a6de60"
server_key_version: "encryptionkey@0"
I20260810 03:18:59.431596 23837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:18:59.450322 23837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:18:59.455448 23837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:18:59.455874 23837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:18:59.457034 23837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:18:59.458905 23837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:18:59.459026 23837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:59.459194 23837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:18:59.459283 23837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:18:59.497006 23837 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:32905
I20260810 03:18:59.497046 23976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:32905 every 8 connection(s)
I20260810 03:18:59.498520 23837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:18:59.502789 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 23837
I20260810 03:18:59.503027 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:18:59.503885 20443 external_mini_cluster.cc:1428] Setting key 0cb1244ece029ea1ed7115468532dd70
Aug 10 03:18:59 dist-test-slave-btw8 krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331939, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:18:59.524832 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:59.501501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:38421 (local address 127.19.246.254:43709)
0810 03:18:59.502039 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 03:18:59.502044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:18:59.503407 (+ 1363us) server_negotiation.cc:407] Connection header received
0810 03:18:59.506000 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:18:59.506005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:18:59.506126 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:18:59.506254 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:18:59.508206 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:59.509375 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:59.511301 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:18:59.512061 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:18:59.512237 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:59.516475 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:18:59.516485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:18:59.516493 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:18:59.516546 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:18:59.520551 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:59.521528 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:59.521537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:59.521544 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:59.521727 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:18:59.522376 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:18:59.522384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:18:59.522390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:18:59.522809 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:18:59.522976 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 03:18:59.524004 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0810 03:18:59.524325 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":286,"threads_started":1}
I20260810 03:18:59.525787 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:18:59.502231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43709 (local address 127.19.246.193:38421)
0810 03:18:59.503185 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0810 03:18:59.503243 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:18:59.505617 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:18:59.506569 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:18:59.506583 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:18:59.507312 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:18:59.508008 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:59.508026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:59.509573 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:59.509577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:59.511104 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:18:59.511119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:18:59.512477 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:18:59.512488 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:18:59.512791 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:18:59.513652 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:18:59.513677 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:18:59.516158 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:18:59.520774 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:59.520783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:59.520799 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:59.521357 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:59.521899 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:18:59.521903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:18:59.521905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:18:59.522206 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:18:59.522972 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:18:59.522999 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:18:59.523524 (+ 525us) client_negotiation.cc:769] Sending connection context
0810 03:18:59.523819 (+ 295us) client_negotiation.cc:240] Negotiation successful
0810 03:18:59.524428 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":364,"threads_started":1}
I20260810 03:18:59.527814 23977 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43709
I20260810 03:18:59.528195 23977 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
I20260810 03:18:59.529125 23977 heartbeater.cc:511] Master 127.19.246.254:43709 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
I20260810 03:18:59.532326 23765 ts_manager.cc:194] Registered new tserver with Master: 64f8d661edc54dc6aa9be591e15ae4ce (127.19.246.193:32905)
I20260810 03:18:59.535041 23765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:38421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:18:59.568465 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43709
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:18:59.932133 23985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:18:59.932577 23985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:18:59.932773 23985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:18:59.940900 23985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:18:59.941123 23985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:18:59.941334 23985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:18:59.952420 23985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:18:59.953953 23985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:18:59.955464 23985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:18:59.972409 23990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:59.972409 23993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:18:59.972498 23991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:18:59.974458 23985 server_base.cc:1034] running on GCE node
I20260810 03:18:59.975265 23985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:18:59.976395 23985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:18:59.977733 23985 hybrid_clock.cc:648] HybridClock initialized: now 1786331939977622 us; error 46 us; skew 500 ppm
I20260810 03:18:59.978117 23985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:18:59 dist-test-slave-btw8 krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331939, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:18:59.982437 23985 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:18:59.984457 23985 webserver.cc:460] Webserver started at http://127.19.246.194:39409/ using document root <none> and password file <none>
I20260810 03:18:59.985317 23985 fs_manager.cc:359] Metadata directory not provided
I20260810 03:18:59.985461 23985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:18:59.985801 23985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:18:59.989231 23985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "badcb0686b4d4d0dad110662e451ce86"
format_stamp: "Formatted at 2026-08-10 03:18:59 on dist-test-slave-btw8"
server_key: "838fd5d135c3aa063eaf46e3a721fc43"
server_key_iv: "8612f49f309767cd0c4862a252181288"
server_key_version: "encryptionkey@0"
I20260810 03:18:59.990226 23985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "badcb0686b4d4d0dad110662e451ce86"
format_stamp: "Formatted at 2026-08-10 03:18:59 on dist-test-slave-btw8"
server_key: "838fd5d135c3aa063eaf46e3a721fc43"
server_key_iv: "8612f49f309767cd0c4862a252181288"
server_key_version: "encryptionkey@0"
I20260810 03:18:59.996644 23985 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 03:19:00.000959 24000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:00.002763 23985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260810 03:19:00.003003 23985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "badcb0686b4d4d0dad110662e451ce86"
format_stamp: "Formatted at 2026-08-10 03:18:59 on dist-test-slave-btw8"
server_key: "838fd5d135c3aa063eaf46e3a721fc43"
server_key_iv: "8612f49f309767cd0c4862a252181288"
server_key_version: "encryptionkey@0"
I20260810 03:19:00.003258 23985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:00.023733 23985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:00.028846 23985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:00.029201 23985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:00.030313 23985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:00.032068 23985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:00.032241 23985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:00.032404 23985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:00.032477 23985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:00.069589 23985 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:39769
I20260810 03:19:00.069628 24125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:39769 every 8 connection(s)
I20260810 03:19:00.071050 23985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:00.075922 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 23985
I20260810 03:19:00.076156 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:19:00.076768 20443 external_mini_cluster.cc:1428] Setting key a9a5fffb1fe9802c14856cc98d0bd669
Aug 10 03:19:00 dist-test-slave-btw8 krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331939, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:00.099222 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:00.074316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:41701 (local address 127.19.246.254:43709)
0810 03:19:00.074847 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:00.074852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:00.076433 (+ 1581us) server_negotiation.cc:407] Connection header received
0810 03:19:00.078630 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:00.078636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:00.078719 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:00.078856 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:00.081571 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.082395 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.085279 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.085811 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.085936 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:00.090778 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:00.090788 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:00.090797 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:00.090854 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:00.095034 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.096264 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.096274 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.096281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.096478 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.096999 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.097007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.097013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.097451 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:00.097629 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:00.098440 (+ 811us) server_negotiation.cc:299] Negotiation successful
0810 03:19:00.098746 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":279,"threads_started":1}
I20260810 03:19:00.100322 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:00.075041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43709 (local address 127.19.246.194:41701)
0810 03:19:00.076210 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:00.076272 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:00.078296 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:00.079188 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:00.079199 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:00.080267 (+ 1068us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:00.081283 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:00.081316 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.082598 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.082607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:00.085026 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:00.085050 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.086101 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.086112 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:00.086492 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:00.087407 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:00.087453 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:00.090431 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:00.095300 (+ 4869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:00.095309 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:00.095327 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:00.096074 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:00.096639 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:00.096643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:00.096645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:00.096866 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:00.097606 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:00.097613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:00.098017 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 03:19:00.098281 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 03:19:00.098936 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":536,"threads_started":1}
I20260810 03:19:00.102355 24126 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43709
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
I20260810 03:19:00.102712 24126 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:00.103664 24126 heartbeater.cc:511] Master 127.19.246.254:43709 requested a full tablet report, sending...
I20260810 03:19:00.105584 23765 ts_manager.cc:194] Registered new tserver with Master: badcb0686b4d4d0dad110662e451ce86 (127.19.246.194:39769)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
I20260810 03:19:00.106786 23765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:41701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:19:00.140391 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43709
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:00.515298 24134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:00.515695 24134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:00.515798 24134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:00.523662 24134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:00.523793 24134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:00.523911 24134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:00.534478 24134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:00.535965 24134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:00.537334 24134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:19:00.539695 23977 heartbeater.cc:503] Master 127.19.246.254:43709 was elected leader, sending a full tablet report...
W20260810 03:19:00.553002 24139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:00.553053 24140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:00.553059 24142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:00.554141 24134 server_base.cc:1034] running on GCE node
I20260810 03:19:00.554805 24134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:00.556125 24134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:00.557403 24134 hybrid_clock.cc:648] HybridClock initialized: now 1786331940557306 us; error 93 us; skew 500 ppm
I20260810 03:19:00.557762 24134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:00 dist-test-slave-btw8 krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331940, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:00.561456 24134 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:19:00.563303 24134 webserver.cc:460] Webserver started at http://127.19.246.195:33583/ using document root <none> and password file <none>
I20260810 03:19:00.564129 24134 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:00.564255 24134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:00.564659 24134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:00.568205 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "5497ac8d46f84afc9942adff8ccfce55"
format_stamp: "Formatted at 2026-08-10 03:19:00 on dist-test-slave-btw8"
server_key: "6d948651848db07434c8e4a3d321cc3c"
server_key_iv: "3681b773e357990749bdbe2d24643b38"
server_key_version: "encryptionkey@0"
I20260810 03:19:00.569089 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "5497ac8d46f84afc9942adff8ccfce55"
format_stamp: "Formatted at 2026-08-10 03:19:00 on dist-test-slave-btw8"
server_key: "6d948651848db07434c8e4a3d321cc3c"
server_key_iv: "3681b773e357990749bdbe2d24643b38"
server_key_version: "encryptionkey@0"
I20260810 03:19:00.575172 24134 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260810 03:19:00.579625 24149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:00.581230 24134 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:00.581440 24134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "5497ac8d46f84afc9942adff8ccfce55"
format_stamp: "Formatted at 2026-08-10 03:19:00 on dist-test-slave-btw8"
server_key: "6d948651848db07434c8e4a3d321cc3c"
server_key_iv: "3681b773e357990749bdbe2d24643b38"
server_key_version: "encryptionkey@0"
I20260810 03:19:00.581682 24134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:00.604103 24134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:00.609547 24134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:00.609922 24134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:00.611090 24134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:00.612805 24134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:00.612898 24134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:00.613096 24134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:00.613175 24134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:00.652315 24134 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:39411
I20260810 03:19:00.652357 24273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:39411 every 8 connection(s)
I20260810 03:19:00.653705 24134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:00.658689 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 24134
I20260810 03:19:00.658881 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:19:00.659416 20443 external_mini_cluster.cc:1428] Setting key 47beac7baea79a5e1ee2ce89f90be616
Aug 10 03:19:00 dist-test-slave-btw8 krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331940, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:00.677840 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:00.656582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:44901 (local address 127.19.246.254:43709)
0810 03:19:00.657154 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:00.657160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:00.658402 (+ 1242us) server_negotiation.cc:407] Connection header received
0810 03:19:00.660812 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:00.660818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:00.660920 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:00.661045 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:00.663139 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.664085 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.665982 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.666437 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.666627 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:00.671166 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:00.671172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:00.671176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:00.671217 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:00.674522 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.675399 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.675404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.675408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.675539 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.676006 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.676011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.676015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.676301 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:00.676461 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:00.677199 (+ 738us) server_negotiation.cc:299] Negotiation successful
0810 03:19:00.677439 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":277,"threads_started":1}
I20260810 03:19:00.678855 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:00.657295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43709 (local address 127.19.246.195:44901)
0810 03:19:00.658195 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:00.658259 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:00.660452 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:00.661335 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:00.661347 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:00.662178 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:00.662938 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:00.662959 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.664263 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.664268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:00.665786 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:00.665801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.666745 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.666753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:00.667025 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:00.667885 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:00.667917 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:00.670831 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:00.674709 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:00.674720 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:00.674744 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:00.675234 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:00.675660 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:00.675665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:00.675669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:00.675871 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:00.676439 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:00.676448 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:00.676828 (+ 380us) client_negotiation.cc:769] Sending connection context
0810 03:19:00.677118 (+ 290us) client_negotiation.cc:240] Negotiation successful
0810 03:19:00.677715 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":371,"threads_started":1}
I20260810 03:19:00.680615 24274 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43709
I20260810 03:19:00.681013 24274 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:00.681964 24274 heartbeater.cc:511] Master 127.19.246.254:43709 requested a full tablet report, sending...
I20260810 03:19:00.683779 23765 ts_manager.cc:194] Registered new tserver with Master: 5497ac8d46f84afc9942adff8ccfce55 (127.19.246.195:39411)
I20260810 03:19:00.684828 23765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:44901
I20260810 03:19:00.688316 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:19:00 dist-test-slave-btw8 krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331940, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 03:19:00 dist-test-slave-btw8 krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331940, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:00.755668 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:00.736714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.19.246.254:43709)
0810 03:19:00.736926 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:00.736932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:00.737276 (+ 344us) server_negotiation.cc:407] Connection header received
0810 03:19:00.737426 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:00.737430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:00.737576 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:00.737687 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:00.739487 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.740353 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.741811 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.742299 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.742458 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:00.748463 (+ 6005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:00.748469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:00.748473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:00.748523 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:00.752061 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.753077 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.753085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.753097 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.753478 (+ 381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.753976 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.753981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.753984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.754415 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:00.754704 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:00.754979 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 03:19:00.755231 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 03:19:00.756395 23765 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38840
I20260810 03:19:00.758591 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23837
Aug 10 03:19:00 dist-test-slave-btw8 krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331940, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:19:00.778122 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:00.759952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.19.246.193:32905)
0810 03:19:00.760577 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:00.760586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:00.760613 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:19:00.760684 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:00.760691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:00.760912 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:00.761067 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:00.762487 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.763731 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.765186 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.765938 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.766061 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:00.771158 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:00.771174 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:00.771191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:00.771252 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:00.774766 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.775709 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.775718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.775721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.775871 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.776302 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.776308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.776312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.776726 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:00.777069 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:00.777397 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 03:19:00.777702 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":302,"threads_started":1}
W20260810 03:19:00.780030 23972 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50356
W20260810 03:19:00.781344 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23837: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:19:00.781589 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 23837
I20260810 03:19:00.801852 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23985
Aug 10 03:19:00 dist-test-slave-btw8 krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331940, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:19:00.820686 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:00.802973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.19.246.194:39769)
0810 03:19:00.803709 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:00.803720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:00.803753 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:19:00.803838 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:00.803847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:00.804134 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:00.804314 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:00.805701 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.806922 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.808588 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.809258 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.809379 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:00.814339 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:00.814357 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:00.814383 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:00.814434 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:00.817546 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.818357 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.818365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.818372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.818514 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.818921 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.818926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.818929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.819309 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:00.819636 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:00.820002 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 03:19:00.820266 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":416,"threads_started":1}
W20260810 03:19:00.822238 24120 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43116
W20260810 03:19:00.823642 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23985: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:19:00.823864 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 23985
I20260810 03:19:00.844429 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 24134
Aug 10 03:19:00 dist-test-slave-btw8 krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331940, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:19:00.865542 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:00.845758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.19.246.195:39411)
0810 03:19:00.846505 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:00.846520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:00.846553 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:19:00.846668 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:00.846681 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:00.846906 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:00.847196 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:00.849160 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.850454 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.852215 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:00.852923 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:00.853050 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:00.858433 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:00.858453 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:00.858470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:00.858519 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:00.861826 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.862818 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.862827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.862830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.862980 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:00.863597 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:00.863602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:00.863604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:00.864104 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:00.864454 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:00.864790 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 03:19:00.865049 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":370,"threads_started":1}
W20260810 03:19:00.867093 24268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43110
W20260810 03:19:00.868557 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 24134: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:19:00.868813 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 24134
I20260810 03:19:00.888984 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 23731
I20260810 03:19:00.890444 23800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:01.003618 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 23731
2026-08-10T03:19:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4183 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:19:01 dist-test-slave-btw8 krb5kdc[24292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:19:01 dist-test-slave-btw8 krb5kdc[24292](info): set up 2 sockets
Aug 10 03:19:01 dist-test-slave-btw8 krb5kdc[24292](info): commencing operation
krb5kdc: starting...
W20260810 03:19:02.260257 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.009s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:19:02 dist-test-slave-btw8 krb5kdc[24292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:19:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:19:02.544523 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:34611
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:02.927712 24307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:02.928088 24307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:02.928196 24307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:02.936832 24307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:02.936981 24307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:02.937038 24307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:02.937093 24307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:02.937140 24307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:02.948598 24307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:02.950222 24307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:02.951867 24307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:02.965559 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:02.965827 24315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:02.965986 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:02.967300 24307 server_base.cc:1034] running on GCE node
I20260810 03:19:02.968470 24307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:02.970363 24307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:02.971679 24307 hybrid_clock.cc:648] HybridClock initialized: now 1786331942971622 us; error 69 us; skew 500 ppm
I20260810 03:19:02.972143 24307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:02 dist-test-slave-btw8 krb5kdc[24292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331942, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:02.976243 24307 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:19:02.978180 24307 webserver.cc:460] Webserver started at http://127.19.246.254:40959/ using document root <none> and password file <none>
I20260810 03:19:02.979041 24307 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:02.979221 24307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:02.979637 24307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:02.983219 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "163bf0c60ed642f893f293b31447e6c3"
format_stamp: "Formatted at 2026-08-10 03:19:02 on dist-test-slave-btw8"
server_key: "32997e6e0f04db59ad315a72e6f1f271"
server_key_iv: "b93f068977f879636e20205eb1a22541"
server_key_version: "encryptionkey@0"
I20260810 03:19:02.984252 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "163bf0c60ed642f893f293b31447e6c3"
format_stamp: "Formatted at 2026-08-10 03:19:02 on dist-test-slave-btw8"
server_key: "32997e6e0f04db59ad315a72e6f1f271"
server_key_iv: "b93f068977f879636e20205eb1a22541"
server_key_version: "encryptionkey@0"
I20260810 03:19:02.990927 24307 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:19:02.995464 24322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:02.997411 24307 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:02.997720 24307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "163bf0c60ed642f893f293b31447e6c3"
format_stamp: "Formatted at 2026-08-10 03:19:02 on dist-test-slave-btw8"
server_key: "32997e6e0f04db59ad315a72e6f1f271"
server_key_iv: "b93f068977f879636e20205eb1a22541"
server_key_version: "encryptionkey@0"
I20260810 03:19:02.998003 24307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:03.026262 24307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:03.031976 24307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:03.032338 24307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:03.054160 24307 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:34611
I20260810 03:19:03.054198 24379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:34611 every 8 connection(s)
I20260810 03:19:03.055823 24307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:03.060262 24380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:03.063756 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 24307
I20260810 03:19:03.064038 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:03.064865 20443 external_mini_cluster.cc:1428] Setting key 18b35444252ef173871b7058ccdbd85b
I20260810 03:19:03.076650 24380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3: Bootstrap starting.
I20260810 03:19:03.081791 24380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:03.084053 24380 log.cc:824] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:19:03 dist-test-slave-btw8 krb5kdc[24292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:03.088249 24380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3: No bootstrap required, opened a new log
I20260810 03:19:03.095043 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:03.067881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.19.246.254:34611)
0810 03:19:03.069096 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:03.069109 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:03.069157 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:19:03.071120 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:03.071150 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:03.071754 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:03.072172 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:03.074391 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:03.076048 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:03.078206 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:03.079048 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:03.079242 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:03.086281 (+ 7039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:03.086291 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:03.086310 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:03.086356 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:03.090010 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:03.090973 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:03.090981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:03.090992 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:03.091207 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:03.091734 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:03.091738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:03.091740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:03.092389 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:03.092724 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:03.093289 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 03:19:03.093830 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":639,"threads_started":1}
I20260810 03:19:03.101092 24380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163bf0c60ed642f893f293b31447e6c3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34611 } }
I20260810 03:19:03.102098 24380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:03.102428 24380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163bf0c60ed642f893f293b31447e6c3, State: Initialized, Role: FOLLOWER
I20260810 03:19:03.103929 24380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163bf0c60ed642f893f293b31447e6c3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34611 } }
I20260810 03:19:03.104465 24380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:03.104702 24380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:03.104945 24380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:03.108014 24380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163bf0c60ed642f893f293b31447e6c3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34611 } }
I20260810 03:19:03.108691 24380 leader_election.cc:302] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 163bf0c60ed642f893f293b31447e6c3; no voters:
I20260810 03:19:03.109546 24380 leader_election.cc:288] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:03.110021 24385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:03.111651 24385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [term 1 LEADER]: Becoming Leader. State: Replica: 163bf0c60ed642f893f293b31447e6c3, State: Running, Role: LEADER
I20260810 03:19:03.112349 24380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:03.112771 24385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163bf0c60ed642f893f293b31447e6c3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34611 } }
I20260810 03:19:03.116683 24387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 163bf0c60ed642f893f293b31447e6c3. Latest consensus state: current_term: 1 leader_uuid: "163bf0c60ed642f893f293b31447e6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163bf0c60ed642f893f293b31447e6c3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34611 } } }
I20260810 03:19:03.116971 24386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "163bf0c60ed642f893f293b31447e6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163bf0c60ed642f893f293b31447e6c3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34611 } } }
I20260810 03:19:03.117429 24387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:03.117586 24386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:03.123153 24393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:03.130137 24393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:03.142127 24393 catalog_manager.cc:1347] Generated new cluster ID: 3843963c0267474aa555b7febd722dff
I20260810 03:19:03.142324 24393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:03.153719 24393 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:03.155225 24393 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:03.164819 24393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 163bf0c60ed642f893f293b31447e6c3: Generated new TSK 0
I20260810 03:19:03.165925 24393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:19:03.244789 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34611
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:03.638907 24408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:03.639259 24408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:03.639457 24408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:03.647392 24408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:03.647550 24408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:03.647707 24408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:03.658435 24408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:03.659938 24408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:03.661459 24408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:03.676506 24415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:03.676565 24414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:03.676558 24417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:03.677706 24408 server_base.cc:1034] running on GCE node
I20260810 03:19:03.678478 24408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:03.679687 24408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:03.680951 24408 hybrid_clock.cc:648] HybridClock initialized: now 1786331943680905 us; error 46 us; skew 500 ppm
I20260810 03:19:03.681342 24408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:03 dist-test-slave-btw8 krb5kdc[24292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331943, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:03.684938 24408 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:19:03.686570 24408 webserver.cc:460] Webserver started at http://127.19.246.193:42345/ using document root <none> and password file <none>
I20260810 03:19:03.687336 24408 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:03.687543 24408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:03.687878 24408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:03.690949 24408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "f5f5a55d765d4429b1823cc0e3c28eb9"
format_stamp: "Formatted at 2026-08-10 03:19:03 on dist-test-slave-btw8"
server_key: "37c07aed0dbb3395aba3fbac11177f42"
server_key_iv: "264eb8f2525b693317e4fe3666196b27"
server_key_version: "encryptionkey@0"
I20260810 03:19:03.691834 24408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "f5f5a55d765d4429b1823cc0e3c28eb9"
format_stamp: "Formatted at 2026-08-10 03:19:03 on dist-test-slave-btw8"
server_key: "37c07aed0dbb3395aba3fbac11177f42"
server_key_iv: "264eb8f2525b693317e4fe3666196b27"
server_key_version: "encryptionkey@0"
I20260810 03:19:03.697978 24408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:19:03.702512 24425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:03.704360 24408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 03:19:03.704640 24408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "f5f5a55d765d4429b1823cc0e3c28eb9"
format_stamp: "Formatted at 2026-08-10 03:19:03 on dist-test-slave-btw8"
server_key: "37c07aed0dbb3395aba3fbac11177f42"
server_key_iv: "264eb8f2525b693317e4fe3666196b27"
server_key_version: "encryptionkey@0"
I20260810 03:19:03.704892 24408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:03.723028 24408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:03.728132 24408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:03.728464 24408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:03.729506 24408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:03.730959 24408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:03.731061 24408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:03.731211 24408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:03.731297 24408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:03.767515 24408 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:42237
I20260810 03:19:03.767611 24551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:42237 every 8 connection(s)
I20260810 03:19:03.768926 24408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:03.769738 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 24408
I20260810 03:19:03.770005 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:19:03.770821 20443 external_mini_cluster.cc:1428] Setting key 1dea50c7279119bf8189d1863b3d5568
Aug 10 03:19:03 dist-test-slave-btw8 krb5kdc[24292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331943, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:03.793062 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:03.771939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:49413 (local address 127.19.246.254:34611)
0810 03:19:03.772406 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:03.772411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:03.773629 (+ 1218us) server_negotiation.cc:407] Connection header received
0810 03:19:03.775658 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:03.775667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:03.775766 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:03.775892 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:03.777606 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:03.778826 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:03.780878 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:03.781687 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:03.781856 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:03.786369 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:03.786376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:03.786379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:03.786419 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:03.789567 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:03.790473 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:03.790478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:03.790480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:03.790612 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:03.791128 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:03.791132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:03.791134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:03.791487 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:03.791659 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:03.792425 (+ 766us) server_negotiation.cc:299] Negotiation successful
0810 03:19:03.792677 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":250,"threads_started":1}
I20260810 03:19:03.794117 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:03.772611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34611 (local address 127.19.246.193:49413)
0810 03:19:03.773468 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:03.773515 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:03.775287 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:03.776153 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:03.776162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:03.776763 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:03.777403 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:03.777422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:03.779045 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:03.779050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:03.780701 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:03.780716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:03.781967 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:03.781978 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:03.782260 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:03.783146 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:03.783193 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:03.786072 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:03.789776 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:03.789787 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:03.789809 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:03.790322 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:03.790730 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:03.790735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:03.790739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:03.790986 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:03.791632 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:03.791641 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:03.792028 (+ 387us) client_negotiation.cc:769] Sending connection context
0810 03:19:03.792305 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 03:19:03.792877 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
I20260810 03:19:03.796031 24552 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34611
I20260810 03:19:03.796360 24552 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:03.797143 24552 heartbeater.cc:511] Master 127.19.246.254:34611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
I20260810 03:19:03.799644 24341 ts_manager.cc:194] Registered new tserver with Master: f5f5a55d765d4429b1823cc0e3c28eb9 (127.19.246.193:42237)
I20260810 03:19:03.801317 24341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:49413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:19:03.833403 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34611
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:04.210472 24560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:04.210796 24560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:04.210964 24560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:04.218883 24560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:04.219008 24560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:04.219188 24560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:04.230085 24560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:04.231680 24560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:04.233153 24560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:04.249547 24566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:04.249643 24565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:04.251204 24568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:04.252379 24560 server_base.cc:1034] running on GCE node
I20260810 03:19:04.253199 24560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:04.254415 24560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:04.255654 24560 hybrid_clock.cc:648] HybridClock initialized: now 1786331944255591 us; error 58 us; skew 500 ppm
I20260810 03:19:04.256017 24560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:04 dist-test-slave-btw8 krb5kdc[24292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331944, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:04.259835 24560 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:19:04.261564 24560 webserver.cc:460] Webserver started at http://127.19.246.194:39637/ using document root <none> and password file <none>
I20260810 03:19:04.262562 24560 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:04.262766 24560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:04.263180 24560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:04.266937 24560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "9847239f77d2473b8750ba1e91b59655"
format_stamp: "Formatted at 2026-08-10 03:19:04 on dist-test-slave-btw8"
server_key: "1623ea135ccb63ecc9a1dde8bc627d06"
server_key_iv: "af14568e8ec9b5ab94dedc1f59e68332"
server_key_version: "encryptionkey@0"
I20260810 03:19:04.268025 24560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "9847239f77d2473b8750ba1e91b59655"
format_stamp: "Formatted at 2026-08-10 03:19:04 on dist-test-slave-btw8"
server_key: "1623ea135ccb63ecc9a1dde8bc627d06"
server_key_iv: "af14568e8ec9b5ab94dedc1f59e68332"
server_key_version: "encryptionkey@0"
I20260810 03:19:04.276885 24560 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260810 03:19:04.282207 24575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:04.284858 24560 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 03:19:04.285171 24560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "9847239f77d2473b8750ba1e91b59655"
format_stamp: "Formatted at 2026-08-10 03:19:04 on dist-test-slave-btw8"
server_key: "1623ea135ccb63ecc9a1dde8bc627d06"
server_key_iv: "af14568e8ec9b5ab94dedc1f59e68332"
server_key_version: "encryptionkey@0"
I20260810 03:19:04.285450 24560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:04.305037 24560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:04.311686 24560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:04.312307 24560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:04.313798 24560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:04.315685 24560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:04.315814 24560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:04.315992 24560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:04.316079 24560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:04.357568 24560 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:46303
I20260810 03:19:04.357699 24703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:46303 every 8 connection(s)
I20260810 03:19:04.359076 24560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:04.362264 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 24560
I20260810 03:19:04.362490 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:19:04.363085 20443 external_mini_cluster.cc:1428] Setting key 3c09c03976e149c6e38bf7c29648572c
Aug 10 03:19:04 dist-test-slave-btw8 krb5kdc[24292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331944, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:04.388573 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:04.362320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:58349 (local address 127.19.246.254:34611)
0810 03:19:04.362917 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:04.362925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:04.364239 (+ 1314us) server_negotiation.cc:407] Connection header received
0810 03:19:04.366975 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:04.366980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:04.367066 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:04.367184 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:04.369854 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:04.370771 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:04.373821 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:04.374318 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:04.374442 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:04.379130 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:04.379141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:04.379150 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:04.379220 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:04.383813 (+ 4593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:04.384900 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:04.384916 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:04.384923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:04.385119 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:04.385809 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:04.385819 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:04.385826 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:04.386340 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:04.386524 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:04.387532 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0810 03:19:04.387862 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"spinlock_wait_cycles":18304,"thread_start_us":301,"threads_started":1}
I20260810 03:19:04.389220 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:04.363016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34611 (local address 127.19.246.194:58349)
0810 03:19:04.364010 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:04.364088 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:04.366602 (+ 2514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:04.367577 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:04.367595 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:04.368580 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:04.369586 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:04.369620 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:04.370974 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:04.370984 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:04.373601 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:04.373624 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:04.374625 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:04.374636 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:04.374909 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:04.375943 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:04.375977 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:04.378753 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:04.384073 (+ 5320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:04.384084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:04.384102 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:04.384715 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:04.385324 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:04.385331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:04.385335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:04.385660 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:04.386584 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:04.386591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:04.387012 (+ 421us) client_negotiation.cc:769] Sending connection context
0810 03:19:04.387317 (+ 305us) client_negotiation.cc:240] Negotiation successful
0810 03:19:04.387881 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":433,"threads_started":1}
I20260810 03:19:04.391055 24704 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34611
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
I20260810 03:19:04.391528 24704 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:04.392449 24704 heartbeater.cc:511] Master 127.19.246.254:34611 requested a full tablet report, sending...
I20260810 03:19:04.394387 24341 ts_manager.cc:194] Registered new tserver with Master: 9847239f77d2473b8750ba1e91b59655 (127.19.246.194:46303)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
I20260810 03:19:04.395699 24341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:58349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:19:04.428041 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34611
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:04.789517 24710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:04.789923 24710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:04.790112 24710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:04.798038 24710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:04.798235 24710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:04.798427 24710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:04.804749 24552 heartbeater.cc:503] Master 127.19.246.254:34611 was elected leader, sending a full tablet report...
I20260810 03:19:04.810011 24710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:04.811589 24710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:04.813164 24710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:04.829336 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:04.829313 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:04.829264 24718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:04.830505 24710 server_base.cc:1034] running on GCE node
I20260810 03:19:04.831305 24710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:04.832412 24710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:04.833712 24710 hybrid_clock.cc:648] HybridClock initialized: now 1786331944833657 us; error 49 us; skew 500 ppm
I20260810 03:19:04.834084 24710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:04 dist-test-slave-btw8 krb5kdc[24292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331944, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:04.842914 24710 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:19:04.845238 24710 webserver.cc:460] Webserver started at http://127.19.246.195:35113/ using document root <none> and password file <none>
I20260810 03:19:04.846093 24710 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:04.846256 24710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:04.846613 24710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:04.850001 24710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "98de3e8d0bb149a6b8d84e05dbada2fb"
format_stamp: "Formatted at 2026-08-10 03:19:04 on dist-test-slave-btw8"
server_key: "ec94a976b3c95d50b09cdd48927da866"
server_key_iv: "ce775986c8b48160609756dec685c9e9"
server_key_version: "encryptionkey@0"
I20260810 03:19:04.850874 24710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "98de3e8d0bb149a6b8d84e05dbada2fb"
format_stamp: "Formatted at 2026-08-10 03:19:04 on dist-test-slave-btw8"
server_key: "ec94a976b3c95d50b09cdd48927da866"
server_key_iv: "ce775986c8b48160609756dec685c9e9"
server_key_version: "encryptionkey@0"
I20260810 03:19:04.856891 24710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 03:19:04.860982 24725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:04.862686 24710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260810 03:19:04.862895 24710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "98de3e8d0bb149a6b8d84e05dbada2fb"
format_stamp: "Formatted at 2026-08-10 03:19:04 on dist-test-slave-btw8"
server_key: "ec94a976b3c95d50b09cdd48927da866"
server_key_iv: "ce775986c8b48160609756dec685c9e9"
server_key_version: "encryptionkey@0"
I20260810 03:19:04.863123 24710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:04.885725 24710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:04.891052 24710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:04.891445 24710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:04.892520 24710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:04.894007 24710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:04.894115 24710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:04.894289 24710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:04.894367 24710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:04.943581 24710 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:44709
I20260810 03:19:04.943779 24851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:44709 every 8 connection(s)
I20260810 03:19:04.945454 24710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:04.947947 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 24710
I20260810 03:19:04.948197 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:19:04.948746 20443 external_mini_cluster.cc:1428] Setting key c6be835c99e3777a9ab6f762b857824c
Aug 10 03:19:04 dist-test-slave-btw8 krb5kdc[24292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331944, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:04.970927 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:04.949184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:50147 (local address 127.19.246.254:34611)
0810 03:19:04.949726 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:04.949731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:04.950988 (+ 1257us) server_negotiation.cc:407] Connection header received
0810 03:19:04.953426 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:04.953431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:04.953507 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:04.953610 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:04.955490 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:04.956384 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:04.958265 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:04.958746 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:04.958866 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:04.963469 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:04.963476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:04.963480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:04.963523 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:04.967229 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:04.968272 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:04.968277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:04.968285 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:04.968423 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:04.968921 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:04.968925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:04.968927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:04.969221 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:04.969397 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:04.970225 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 03:19:04.970494 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":260,"threads_started":1}
I20260810 03:19:04.972092 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:04.949873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34611 (local address 127.19.246.195:50147)
0810 03:19:04.950824 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:04.950875 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:04.953117 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:04.953932 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:04.953942 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:04.954565 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:04.955223 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:04.955248 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:04.956557 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:04.956561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:04.958073 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:04.958088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:04.959000 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:04.959006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:04.959281 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:04.960178 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:04.960203 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:04.962937 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:04.967445 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:04.967456 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:04.967484 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:04.968134 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:04.968549 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:04.968554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:04.968558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:04.968787 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:04.969361 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:04.969370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:04.969808 (+ 438us) client_negotiation.cc:769] Sending connection context
0810 03:19:04.970109 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 03:19:04.970745 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":418,"threads_started":1}
I20260810 03:19:04.973946 24852 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34611
I20260810 03:19:04.974344 24852 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:04.975214 24852 heartbeater.cc:511] Master 127.19.246.254:34611 requested a full tablet report, sending...
I20260810 03:19:04.977177 24341 ts_manager.cc:194] Registered new tserver with Master: 98de3e8d0bb149a6b8d84e05dbada2fb (127.19.246.195:44709)
I20260810 03:19:04.978710 24341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:50147
I20260810 03:19:04.978804 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:19:05 dist-test-slave-btw8 krb5kdc[24292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:19:05 dist-test-slave-btw8 krb5kdc[24292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:19:05.047566 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:05.027985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.19.246.193:42237)
0810 03:19:05.028713 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:05.028721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:05.028753 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:19:05.028819 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:05.028826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:05.029075 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:05.029224 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:05.031184 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.032491 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.033901 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.034525 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.034701 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:05.039829 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:05.039847 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:05.039863 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:05.039909 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:05.043467 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.044471 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.044518 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.044521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.044763 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.045415 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.045420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.045422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.046002 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:05.046322 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:05.046782 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 03:19:05.047033 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":398,"threads_started":1}
Aug 10 03:19:05 dist-test-slave-btw8 krb5kdc[24292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:05.071647 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:05.053421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.19.246.254:34611)
0810 03:19:05.053695 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:05.053704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:05.053963 (+ 259us) server_negotiation.cc:407] Connection header received
0810 03:19:05.054214 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:05.054219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:05.054302 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:05.054429 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:05.056085 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.056993 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.058487 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.058982 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.059151 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:05.063775 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:05.063782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:05.063785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:05.063836 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:05.067665 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.068960 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.068969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.069040 (+ 71us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.069364 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.069870 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.069875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.069877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.070291 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:05.070613 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:05.070975 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 03:19:05.071207 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260810 03:19:05.072510 24341 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44914
I20260810 03:19:05.073975 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 24408
I20260810 03:19:05.090457 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:05.075099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.19.246.193:42237)
0810 03:19:05.075467 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:05.075473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:05.075747 (+ 274us) server_negotiation.cc:407] Connection header received
0810 03:19:05.075929 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:05.075934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:05.076008 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:05.076104 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:05.077772 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.078523 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.080025 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.080488 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.080616 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:05.083899 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:05.083905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:05.083909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:05.083951 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:05.087045 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.088330 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.088335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.088337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.088503 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.088984 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.088988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.088991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.089292 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:05.089509 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:05.089887 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 03:19:05.090138 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260810 03:19:05.091126 24547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:05.188055 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 24408
I20260810 03:19:05.210078 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 24560
Aug 10 03:19:05 dist-test-slave-btw8 krb5kdc[24292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:19:05.231106 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:05.211042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.19.246.194:46303)
0810 03:19:05.211808 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:05.211816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:05.211845 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:19:05.211915 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:05.211922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:05.212154 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:05.212350 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:05.214425 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.216079 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.217728 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.218479 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.218704 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:05.224124 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:05.224167 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:05.224184 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:05.224232 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:05.227717 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.228784 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.228793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.228795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.228953 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.229471 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.229475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.229478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.229879 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:05.230170 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:05.230559 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 03:19:05.230789 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":432,"threads_started":1}
I20260810 03:19:05.232695 24699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:05.322821 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 24560
I20260810 03:19:05.342705 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 24710
Aug 10 03:19:05 dist-test-slave-btw8 krb5kdc[24292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:19:05.362914 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:05.344229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.19.246.195:44709)
0810 03:19:05.344972 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:05.344980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:05.345012 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:19:05.345074 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:05.345081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:05.345329 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:05.345466 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:05.346915 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.348161 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.349729 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:05.350428 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:05.350542 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:05.355886 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:05.355905 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:05.355922 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:05.355976 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:05.359262 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.360445 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.360453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.360456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.360619 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:05.361160 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:05.361164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:05.361166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:05.361569 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:05.361861 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:05.362239 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 03:19:05.362472 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":410,"threads_started":1}
I20260810 03:19:05.364584 24846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:05.453016 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 24710
I20260810 03:19:05.472905 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 24307
I20260810 03:19:05.474563 24373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:05.586894 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 24307
2026-08-10T03:19:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4584 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:19:05 dist-test-slave-btw8 krb5kdc[24874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:19:05 dist-test-slave-btw8 krb5kdc[24874](info): set up 2 sockets
Aug 10 03:19:05 dist-test-slave-btw8 krb5kdc[24874](info): commencing operation
krb5kdc: starting...
W20260810 03:19:06.842027 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:19:07 dist-test-slave-btw8 krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:19:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:19:07.128602 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40143
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:07.500999 24888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:07.501364 24888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:07.501541 24888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:07.509991 24888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:07.510161 24888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:07.510212 24888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:07.510352 24888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:07.510439 24888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:07.522771 24888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:40143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:07.524317 24888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:07.525787 24888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:07.539824 24893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:07.539886 24896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:07.539883 24894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:07.540894 24888 server_base.cc:1034] running on GCE node
I20260810 03:19:07.542196 24888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:07.544071 24888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:07.545377 24888 hybrid_clock.cc:648] HybridClock initialized: now 1786331947545328 us; error 42 us; skew 500 ppm
I20260810 03:19:07.545943 24888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:07 dist-test-slave-btw8 krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331947, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:07.549748 24888 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:19:07.551291 24888 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:07.551491 24888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:07.551869 24888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:07.555430 24888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "8da7da0b6a7d47b59d1e66a975776891"
format_stamp: "Formatted at 2026-08-10 03:19:07 on dist-test-slave-btw8"
server_key: "da90fe0e188f60c1ea6619059c1550d7"
server_key_iv: "fa0764efded09feaee5df447595f473f"
server_key_version: "encryptionkey@0"
I20260810 03:19:07.556529 24888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "8da7da0b6a7d47b59d1e66a975776891"
format_stamp: "Formatted at 2026-08-10 03:19:07 on dist-test-slave-btw8"
server_key: "da90fe0e188f60c1ea6619059c1550d7"
server_key_iv: "fa0764efded09feaee5df447595f473f"
server_key_version: "encryptionkey@0"
I20260810 03:19:07.563491 24888 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260810 03:19:07.568255 24907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:07.570053 24888 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:19:07.570324 24888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "8da7da0b6a7d47b59d1e66a975776891"
format_stamp: "Formatted at 2026-08-10 03:19:07 on dist-test-slave-btw8"
server_key: "da90fe0e188f60c1ea6619059c1550d7"
server_key_iv: "fa0764efded09feaee5df447595f473f"
server_key_version: "encryptionkey@0"
I20260810 03:19:07.570629 24888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:07.610843 24888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:07.616386 24888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:07.616832 24888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:07.640261 24888 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:40143
I20260810 03:19:07.640281 24964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:40143 every 8 connection(s)
I20260810 03:19:07.641481 24888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:07.645738 24965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:07.648844 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 24888
I20260810 03:19:07.649118 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:07.649837 20443 external_mini_cluster.cc:1428] Setting key f0bad42432a54aebc04c332fb63f7afd
I20260810 03:19:07.660768 24965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891: Bootstrap starting.
I20260810 03:19:07.664414 24965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:07.665620 24965 log.cc:824] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:19:07 dist-test-slave-btw8 krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:07.669440 24965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891: No bootstrap required, opened a new log
I20260810 03:19:07.677345 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:07.652535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.19.246.254:40143)
0810 03:19:07.653655 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:07.653668 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:07.653717 (+ 49us) server_negotiation.cc:407] Connection header received
0810 03:19:07.655761 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:07.655792 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:07.656319 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:07.656744 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:07.659131 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:07.660799 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:07.662437 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:07.663281 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:07.663586 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:07.668512 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:07.668520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:07.668537 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:07.668583 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:07.672383 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:07.673294 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:07.673304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:07.673313 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:07.673519 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:07.673963 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:07.673970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:07.673977 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:07.674685 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:07.675099 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:07.675659 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 03:19:07.676176 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":590,"threads_started":1}
I20260810 03:19:07.679275 24965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7da0b6a7d47b59d1e66a975776891" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40143 } }
I20260810 03:19:07.679813 24965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:07.679952 24965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da7da0b6a7d47b59d1e66a975776891, State: Initialized, Role: FOLLOWER
I20260810 03:19:07.680639 24965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7da0b6a7d47b59d1e66a975776891" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40143 } }
I20260810 03:19:07.680908 24965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:07.681070 24965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:07.681275 24965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:07.683883 24965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7da0b6a7d47b59d1e66a975776891" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40143 } }
I20260810 03:19:07.684434 24965 leader_election.cc:302] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8da7da0b6a7d47b59d1e66a975776891; no voters:
I20260810 03:19:07.685108 24965 leader_election.cc:288] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:07.685271 24970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:07.686103 24970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [term 1 LEADER]: Becoming Leader. State: Replica: 8da7da0b6a7d47b59d1e66a975776891, State: Running, Role: LEADER
I20260810 03:19:07.686774 24970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7da0b6a7d47b59d1e66a975776891" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40143 } }
I20260810 03:19:07.687644 24965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:07.690683 24971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da7da0b6a7d47b59d1e66a975776891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7da0b6a7d47b59d1e66a975776891" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40143 } } }
I20260810 03:19:07.691226 24971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:07.691023 24972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da7da0b6a7d47b59d1e66a975776891. Latest consensus state: current_term: 1 leader_uuid: "8da7da0b6a7d47b59d1e66a975776891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7da0b6a7d47b59d1e66a975776891" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40143 } } }
I20260810 03:19:07.691675 24972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:07.695437 24979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:07.701671 24979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:07.713830 24979 catalog_manager.cc:1347] Generated new cluster ID: 17944442057d46a694399749b3286681
I20260810 03:19:07.713975 24979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:07.728953 24979 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:07.731294 24979 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:07.745189 24979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891: Generated new TSK 0
I20260810 03:19:07.746486 24979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:19:07.823879 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40143
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:08.238214 24991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:08.238767 24991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:08.239013 24991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:08.248555 24991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:08.248704 24991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:08.248907 24991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:08.260685 24991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:08.262226 24991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:08.263792 24991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:08.283239 24996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:08.283314 24998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:08.283293 25000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:08.284607 24991 server_base.cc:1034] running on GCE node
I20260810 03:19:08.285490 24991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:08.286679 24991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:08.287887 24991 hybrid_clock.cc:648] HybridClock initialized: now 1786331948287824 us; error 53 us; skew 500 ppm
I20260810 03:19:08.288378 24991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:08 dist-test-slave-btw8 krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331948, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:08.292341 24991 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:19:08.293985 24991 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:08.294100 24991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:08.294492 24991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:08.298043 24991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "7485e07147114b36a42378ef3c130dc8"
format_stamp: "Formatted at 2026-08-10 03:19:08 on dist-test-slave-btw8"
server_key: "4a5b6bb3581d36aea9d96590158b305e"
server_key_iv: "b1c02d1425bc70b736aa5eb94493cc8b"
server_key_version: "encryptionkey@0"
I20260810 03:19:08.299022 24991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "7485e07147114b36a42378ef3c130dc8"
format_stamp: "Formatted at 2026-08-10 03:19:08 on dist-test-slave-btw8"
server_key: "4a5b6bb3581d36aea9d96590158b305e"
server_key_iv: "b1c02d1425bc70b736aa5eb94493cc8b"
server_key_version: "encryptionkey@0"
I20260810 03:19:08.305472 24991 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:19:08.309875 25008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:08.311789 24991 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:08.312039 24991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "7485e07147114b36a42378ef3c130dc8"
format_stamp: "Formatted at 2026-08-10 03:19:08 on dist-test-slave-btw8"
server_key: "4a5b6bb3581d36aea9d96590158b305e"
server_key_iv: "b1c02d1425bc70b736aa5eb94493cc8b"
server_key_version: "encryptionkey@0"
I20260810 03:19:08.312325 24991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:08.335846 24991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:08.341837 24991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:08.342324 24991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:08.343775 24991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:08.345752 24991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:08.345935 24991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:08.346107 24991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:08.346220 24991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:08.388396 24991 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:33021
I20260810 03:19:08.388415 25134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:33021 every 8 connection(s)
I20260810 03:19:08.389559 24991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:08.399142 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 24991
I20260810 03:19:08.399479 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:19:08.400247 20443 external_mini_cluster.cc:1428] Setting key 6071419972371c8483f34fba3fa11a74
Aug 10 03:19:08 dist-test-slave-btw8 krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331948, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:08.418262 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:08.392940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:46137 (local address 127.19.246.254:40143)
0810 03:19:08.393644 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:08.393652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:08.395411 (+ 1759us) server_negotiation.cc:407] Connection header received
0810 03:19:08.397632 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:08.397638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:08.397751 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:08.397883 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:08.399847 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:08.400702 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:08.402786 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:08.403218 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:08.403387 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:08.408747 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:08.408753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:08.408756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:08.408802 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:08.413233 (+ 4431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:08.414876 (+ 1643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:08.414882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:08.414885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:08.415075 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:08.415687 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:08.415691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:08.415694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:08.415993 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:08.416198 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:08.417067 (+ 869us) server_negotiation.cc:299] Negotiation successful
0810 03:19:08.417788 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":405,"threads_started":1}
I20260810 03:19:08.419463 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:08.394131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40143 (local address 127.19.246.193:46137)
0810 03:19:08.395166 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:08.395216 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:08.397260 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:08.398171 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:08.398181 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:08.398808 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:08.399624 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:08.399649 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:08.400874 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:08.400880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:08.402609 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:08.402624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:08.403531 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:08.403540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:08.403796 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:08.404847 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:08.404890 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:08.408387 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:08.413888 (+ 5501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:08.413898 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:08.413916 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:08.414694 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:08.415216 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:08.415222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:08.415225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:08.415532 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:08.416156 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:08.416166 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:08.416596 (+ 430us) client_negotiation.cc:769] Sending connection context
0810 03:19:08.416901 (+ 305us) client_negotiation.cc:240] Negotiation successful
0810 03:19:08.417818 (+ 917us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":501,"threads_started":1}
I20260810 03:19:08.421571 25135 heartbeater.cc:347] Connected to a master server at 127.19.246.254:40143
I20260810 03:19:08.421999 25135 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:08.422849 25135 heartbeater.cc:511] Master 127.19.246.254:40143 requested a full tablet report, sending...
I20260810 03:19:08.425346 24927 ts_manager.cc:194] Registered new tserver with Master: 7485e07147114b36a42378ef3c130dc8 (127.19.246.193:33021)
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
I20260810 03:19:08.426923 24927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:46137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:19:08.465389 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40143
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:08.845077 25143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:08.845418 25143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:08.845522 25143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:08.854180 25143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:08.854329 25143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:08.854455 25143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:08.868804 25143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:08.870301 25143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:08.871888 25143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:08.887971 25148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:08.887962 25149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:08.888398 25152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:08.889031 25143 server_base.cc:1034] running on GCE node
I20260810 03:19:08.889843 25143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:08.891026 25143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:08.892308 25143 hybrid_clock.cc:648] HybridClock initialized: now 1786331948892227 us; error 99 us; skew 500 ppm
I20260810 03:19:08.892844 25143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:08 dist-test-slave-btw8 krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331948, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:08.897233 25143 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:19:08.898836 25143 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:08.899008 25143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:08.899470 25143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:08.903040 25143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "40bbaee3931545058150a8169791abd2"
format_stamp: "Formatted at 2026-08-10 03:19:08 on dist-test-slave-btw8"
server_key: "91fa3f2d3e638468d193ae7071f4c6fd"
server_key_iv: "2d023a5ae5b4116ccf2176aab245a520"
server_key_version: "encryptionkey@0"
I20260810 03:19:08.904116 25143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "40bbaee3931545058150a8169791abd2"
format_stamp: "Formatted at 2026-08-10 03:19:08 on dist-test-slave-btw8"
server_key: "91fa3f2d3e638468d193ae7071f4c6fd"
server_key_iv: "2d023a5ae5b4116ccf2176aab245a520"
server_key_version: "encryptionkey@0"
I20260810 03:19:08.910735 25143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260810 03:19:08.915330 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:08.917852 25143 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:08.918143 25143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "40bbaee3931545058150a8169791abd2"
format_stamp: "Formatted at 2026-08-10 03:19:08 on dist-test-slave-btw8"
server_key: "91fa3f2d3e638468d193ae7071f4c6fd"
server_key_iv: "2d023a5ae5b4116ccf2176aab245a520"
server_key_version: "encryptionkey@0"
I20260810 03:19:08.918447 25143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:08.954270 25143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:08.960070 25143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:08.960448 25143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:08.962200 25143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:08.964265 25143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:08.964421 25143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:08.964579 25143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:08.964690 25143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:09.009114 25143 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:33339
I20260810 03:19:09.009224 25282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:33339 every 8 connection(s)
I20260810 03:19:09.010138 25143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:09.016683 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 25143
I20260810 03:19:09.016925 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:19:09.017563 20443 external_mini_cluster.cc:1428] Setting key bbd015071449ae42fbb9845a5bdeecd7
Aug 10 03:19:09 dist-test-slave-btw8 krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331948, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:09.036504 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.013748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:39249 (local address 127.19.246.254:40143)
0810 03:19:09.014515 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.014521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.015417 (+ 896us) server_negotiation.cc:407] Connection header received
0810 03:19:09.017722 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.017728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.017835 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.017987 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:09.019964 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.020699 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.022409 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.022933 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.023126 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.027888 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:09.027899 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:09.027903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:09.027946 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:09.032241 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:09.033260 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:09.033268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:09.033274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:09.033450 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:09.034030 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:09.034037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:09.034043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:09.034504 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:09.034702 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.035722 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.036016 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":489,"threads_started":1}
I20260810 03:19:09.037766 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.014380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40143 (local address 127.19.246.194:39249)
0810 03:19:09.015203 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:09.015246 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:09.017403 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:09.018265 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:09.018275 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:09.018981 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:09.019774 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.019797 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.020893 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.020898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.022237 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.022252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.023207 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.023215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.023545 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.024427 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:09.024458 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:09.027584 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:09.032480 (+ 4896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:09.032489 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:09.032506 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:09.033081 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:09.033620 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:09.033624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:09.033627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:09.033881 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:09.034730 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:09.034737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:09.035206 (+ 469us) client_negotiation.cc:769] Sending connection context
0810 03:19:09.035589 (+ 383us) client_negotiation.cc:240] Negotiation successful
0810 03:19:09.036061 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":348,"threads_started":1}
I20260810 03:19:09.040087 25283 heartbeater.cc:347] Connected to a master server at 127.19.246.254:40143
I20260810 03:19:09.040485 25283 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:09.041473 25283 heartbeater.cc:511] Master 127.19.246.254:40143 requested a full tablet report, sending...
I20260810 03:19:09.044009 24927 ts_manager.cc:194] Registered new tserver with Master: 40bbaee3931545058150a8169791abd2 (127.19.246.194:33339)
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
I20260810 03:19:09.045138 24927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:39249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:19:09.089272 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40143
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:09.431298 25135 heartbeater.cc:503] Master 127.19.246.254:40143 was elected leader, sending a full tablet report...
W20260810 03:19:09.489878 25290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:09.490305 25290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:09.490517 25290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:09.499305 25290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:09.499580 25290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:09.499776 25290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:09.511564 25290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:09.513285 25290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:09.514894 25290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:09.530416 25301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:09.530491 25296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:09.530491 25298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:09.531652 25290 server_base.cc:1034] running on GCE node
I20260810 03:19:09.532495 25290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:09.533638 25290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:09.534895 25290 hybrid_clock.cc:648] HybridClock initialized: now 1786331949534825 us; error 65 us; skew 500 ppm
I20260810 03:19:09.535539 25290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:09 dist-test-slave-btw8 krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331949, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:09.542188 25290 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:19:09.544310 25290 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:09.544575 25290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:09.545122 25290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:09.549793 25290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "38bc27c659d546f59d46ab1a57b20dcf"
format_stamp: "Formatted at 2026-08-10 03:19:09 on dist-test-slave-btw8"
server_key: "97e6cfb6f0e8b6eb4ae93e41cf05df2c"
server_key_iv: "140405c0af82d720ee37326aa1ad877c"
server_key_version: "encryptionkey@0"
I20260810 03:19:09.551677 25290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "38bc27c659d546f59d46ab1a57b20dcf"
format_stamp: "Formatted at 2026-08-10 03:19:09 on dist-test-slave-btw8"
server_key: "97e6cfb6f0e8b6eb4ae93e41cf05df2c"
server_key_iv: "140405c0af82d720ee37326aa1ad877c"
server_key_version: "encryptionkey@0"
I20260810 03:19:09.561689 25290 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260810 03:19:09.568907 25308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:09.571497 25290 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260810 03:19:09.571941 25290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "38bc27c659d546f59d46ab1a57b20dcf"
format_stamp: "Formatted at 2026-08-10 03:19:09 on dist-test-slave-btw8"
server_key: "97e6cfb6f0e8b6eb4ae93e41cf05df2c"
server_key_iv: "140405c0af82d720ee37326aa1ad877c"
server_key_version: "encryptionkey@0"
I20260810 03:19:09.572281 25290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:09.618708 25290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:09.626317 25290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:09.626797 25290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:09.628154 25290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:09.630088 25290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:09.630257 25290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:09.630434 25290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:09.630537 25290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:09.675414 25290 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:38663
I20260810 03:19:09.675518 25433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:38663 every 8 connection(s)
I20260810 03:19:09.676582 25290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:09.682775 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 25290
I20260810 03:19:09.683009 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:19:09.683923 20443 external_mini_cluster.cc:1428] Setting key bdcce59cdac29cc160c3146be52ff506
Aug 10 03:19:09 dist-test-slave-btw8 krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331949, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:09.705410 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.680087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:39241 (local address 127.19.246.254:40143)
0810 03:19:09.680625 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.680631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.681840 (+ 1209us) server_negotiation.cc:407] Connection header received
0810 03:19:09.684519 (+ 2679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.684526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.684639 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.684766 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:09.686777 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.688105 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.691014 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.691851 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.691981 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.696704 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:09.696711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:09.696716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:09.696765 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:09.701017 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:09.702371 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:09.702376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:09.702378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:09.702526 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:09.703033 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:09.703036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:09.703038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:09.703463 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:09.703581 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.704748 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.705085 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":242,"threads_started":1}
I20260810 03:19:09.706450 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.680705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40143 (local address 127.19.246.195:39241)
0810 03:19:09.681621 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:09.681666 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:09.684163 (+ 2497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:09.685143 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:09.685156 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:09.685813 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:09.686532 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.686562 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.688527 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.688538 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.690514 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.690539 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.692183 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.692192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.692444 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.693470 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:09.693497 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:09.696346 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:09.701555 (+ 5209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:09.701565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:09.701583 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:09.702240 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:09.702657 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:09.702660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:09.702663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:09.702923 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:09.703596 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:09.703603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:09.704142 (+ 539us) client_negotiation.cc:769] Sending connection context
0810 03:19:09.704549 (+ 407us) client_negotiation.cc:240] Negotiation successful
0810 03:19:09.705057 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":371,"threads_started":1}
I20260810 03:19:09.708246 25434 heartbeater.cc:347] Connected to a master server at 127.19.246.254:40143
I20260810 03:19:09.708551 25434 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:09.709265 25434 heartbeater.cc:511] Master 127.19.246.254:40143 requested a full tablet report, sending...
I20260810 03:19:09.711036 24927 ts_manager.cc:194] Registered new tserver with Master: 38bc27c659d546f59d46ab1a57b20dcf (127.19.246.195:38663)
I20260810 03:19:09.712157 24927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:39241
I20260810 03:19:09.714910 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:19:09.737965 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.720953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.19.246.254:40143)
0810 03:19:09.721169 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.721174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.721764 (+ 590us) server_negotiation.cc:407] Connection header received
0810 03:19:09.721939 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.721944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.722024 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.722230 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:09.724442 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.725462 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.726966 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.727624 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.727769 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.730754 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:09.730760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:09.730764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:09.730810 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:09.734256 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:09.735203 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:09.735208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:09.735211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:09.735442 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:09.736089 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:09.736093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:09.736095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:09.736642 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:09.736876 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.737372 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.737607 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:19:09.745879 24927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:19:09.754678 24927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:09.782008 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.770237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33021 (local address 127.0.0.1:43066)
0810 03:19:09.771189 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:09.771273 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:09.771514 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:09.772693 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:09.772701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:09.772737 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:09.773181 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.773199 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.775728 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.775735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.778380 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.778397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.780870 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.780876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.781121 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.781178 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 03:19:09.781347 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 03:19:09.781585 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":433,"threads_started":1}
I20260810 03:19:09.782985 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.770493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.19.246.193:33021)
0810 03:19:09.771443 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.771454 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.771491 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:19:09.771752 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.771761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.772079 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.772309 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:09.773470 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.775459 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.778676 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.780607 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.780877 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.781093 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.782366 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.782621 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":466,"threads_started":1}
I20260810 03:19:09.783584 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.771767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:38663 (local address 127.0.0.1:57932)
0810 03:19:09.772386 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:09.772407 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:09.772541 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:09.774775 (+ 2234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:09.774780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:09.774811 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:09.775220 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.775232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.778215 (+ 2983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.778231 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.780315 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.780329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.782633 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.782640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.782842 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.782891 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 03:19:09.782967 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:19:09.783148 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":1,"thread_start_us":322,"threads_started":1}
I20260810 03:19:09.784808 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.772781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.19.246.194:33339)
0810 03:19:09.774945 (+ 2164us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.774958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.774997 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:19:09.775081 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.775087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.775291 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.775572 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:09.776269 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.778609 (+ 2340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.781707 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.783552 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.783776 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.783981 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.784232 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.784444 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":1428,"threads_started":1}
I20260810 03:19:09.784838 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.772256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.19.246.195:38663)
0810 03:19:09.773221 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.773229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.773259 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:19:09.773387 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.773398 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.773688 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.774642 (+ 954us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:09.775536 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.777586 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.780513 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.782083 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.784202 (+ 2119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.784380 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.784456 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.784594 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":620,"threads_started":1}
I20260810 03:19:09.785544 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.773146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:33339 (local address 127.0.0.1:50312)
0810 03:19:09.773810 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:09.773832 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:09.773951 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:09.775568 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:09.775573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:09.775601 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:09.776053 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.776065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.778749 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.778757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.781456 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.781474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.783817 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.783825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.784010 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.784054 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:19:09.784179 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 03:19:09.785153 (+ 974us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":1,"thread_start_us":344,"threads_started":1}
I20260810 03:19:09.788131 25063 tablet_service.cc:1467] Processing CreateTablet for tablet fd8bc698ccbf40f4ac09f129a43af2a6 (DEFAULT_TABLE table=test-table [id=c48bb10fa5614c05801dd5a85ba6fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:09.789103 25211 tablet_service.cc:1467] Processing CreateTablet for tablet fd8bc698ccbf40f4ac09f129a43af2a6 (DEFAULT_TABLE table=test-table [id=c48bb10fa5614c05801dd5a85ba6fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:09.789064 25362 tablet_service.cc:1467] Processing CreateTablet for tablet fd8bc698ccbf40f4ac09f129a43af2a6 (DEFAULT_TABLE table=test-table [id=c48bb10fa5614c05801dd5a85ba6fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:09.790053 25063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8bc698ccbf40f4ac09f129a43af2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:09.790383 25211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8bc698ccbf40f4ac09f129a43af2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:09.790639 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8bc698ccbf40f4ac09f129a43af2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:09.803442 25456 tablet_bootstrap.cc:492] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Bootstrap starting.
I20260810 03:19:09.806792 25458 tablet_bootstrap.cc:492] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Bootstrap starting.
I20260810 03:19:09.809661 25456 tablet_bootstrap.cc:654] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:09.810731 25458 tablet_bootstrap.cc:654] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:09.811182 25456 log.cc:824] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:09.812214 25458 log.cc:824] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:09.813052 25457 tablet_bootstrap.cc:492] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Bootstrap starting.
I20260810 03:19:09.818564 25456 tablet_bootstrap.cc:492] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: No bootstrap required, opened a new log
I20260810 03:19:09.818764 25457 tablet_bootstrap.cc:654] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:09.818954 25456 ts_tablet_manager.cc:1397] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260810 03:19:09.820641 25457 log.cc:824] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:09.824916 25457 tablet_bootstrap.cc:492] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: No bootstrap required, opened a new log
I20260810 03:19:09.824916 25458 tablet_bootstrap.cc:492] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: No bootstrap required, opened a new log
I20260810 03:19:09.825276 25457 ts_tablet_manager.cc:1397] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260810 03:19:09.825276 25458 ts_tablet_manager.cc:1397] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20260810 03:19:09.829144 25456 raft_consensus.cc:348] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.829668 25456 raft_consensus.cc:374] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:09.829829 25456 raft_consensus.cc:729] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bbaee3931545058150a8169791abd2, State: Initialized, Role: FOLLOWER
I20260810 03:19:09.830679 25456 consensus_queue.cc:260] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.832108 25283 heartbeater.cc:503] Master 127.19.246.254:40143 was elected leader, sending a full tablet report...
I20260810 03:19:09.833177 25456 ts_tablet_manager.cc:1428] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260810 03:19:09.837064 25458 raft_consensus.cc:348] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.837066 25457 raft_consensus.cc:348] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.837605 25457 raft_consensus.cc:374] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:09.837605 25458 raft_consensus.cc:374] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:09.837728 25458 raft_consensus.cc:729] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38bc27c659d546f59d46ab1a57b20dcf, State: Initialized, Role: FOLLOWER
I20260810 03:19:09.837772 25457 raft_consensus.cc:729] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7485e07147114b36a42378ef3c130dc8, State: Initialized, Role: FOLLOWER
I20260810 03:19:09.838442 25457 consensus_queue.cc:260] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.838442 25458 consensus_queue.cc:260] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.840451 25434 heartbeater.cc:503] Master 127.19.246.254:40143 was elected leader, sending a full tablet report...
I20260810 03:19:09.840814 25457 ts_tablet_manager.cc:1428] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260810 03:19:09.842234 25458 ts_tablet_manager.cc:1428] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
W20260810 03:19:09.892657 25137 tablet.cc:2369] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:19:09.900422 25463 raft_consensus.cc:482] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:09.900949 25463 raft_consensus.cc:504] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.902995 25463 leader_election.cc:288] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38bc27c659d546f59d46ab1a57b20dcf (127.19.246.195:38663), 40bbaee3931545058150a8169791abd2 (127.19.246.194:33339)
I20260810 03:19:09.914400 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.903755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:50213 (local address 127.19.246.195:38663)
0810 03:19:09.904223 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.904231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.904797 (+ 566us) server_negotiation.cc:407] Connection header received
0810 03:19:09.905146 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.905154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.905255 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.905330 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:09.907298 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.908624 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.912203 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.913160 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.913362 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.913466 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.913931 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.914142 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260810 03:19:09.914392 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.905386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:59687 (local address 127.19.246.194:33339)
0810 03:19:09.905609 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.905615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.906737 (+ 1122us) server_negotiation.cc:407] Connection header received
0810 03:19:09.906961 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.906968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.907051 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.907151 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:09.908169 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.909317 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.912227 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.913203 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.913432 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.913522 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.913906 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.914075 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:19:09.914557 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.904872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:33339 (local address 127.19.246.193:59687)
0810 03:19:09.906544 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:09.906593 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:09.906786 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:09.907596 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:09.907601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:09.907630 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:09.907978 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.907987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.909520 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.909527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.911943 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.911961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.913521 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.913528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.913685 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.913740 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 03:19:09.913827 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 03:19:09.914010 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"mutex_wait_us":44,"thread_start_us":1238,"threads_started":1}
I20260810 03:19:09.914604 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.903470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:38663 (local address 127.19.246.193:50213)
0810 03:19:09.904688 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:09.904710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:09.904932 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:09.905570 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:09.905575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:09.905605 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:09.907015 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.907032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.908767 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.908777 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.911906 (+ 3129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:09.911931 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.913472 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.913481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:09.913722 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.913748 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:19:09.913882 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 03:19:09.914048 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":167,"thread_start_us":879,"threads_started":1}
I20260810 03:19:09.916141 25234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8bc698ccbf40f4ac09f129a43af2a6" candidate_uuid: "7485e07147114b36a42378ef3c130dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bbaee3931545058150a8169791abd2" is_pre_election: true
I20260810 03:19:09.916174 25385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8bc698ccbf40f4ac09f129a43af2a6" candidate_uuid: "7485e07147114b36a42378ef3c130dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38bc27c659d546f59d46ab1a57b20dcf" is_pre_election: true
I20260810 03:19:09.916832 25385 raft_consensus.cc:2454] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7485e07147114b36a42378ef3c130dc8 in term 0.
I20260810 03:19:09.916857 25234 raft_consensus.cc:2454] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7485e07147114b36a42378ef3c130dc8 in term 0.
I20260810 03:19:09.917857 25012 leader_election.cc:302] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38bc27c659d546f59d46ab1a57b20dcf, 7485e07147114b36a42378ef3c130dc8; no voters:
I20260810 03:19:09.918674 25463 raft_consensus.cc:2781] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:09.918906 25463 raft_consensus.cc:482] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:09.919044 25463 raft_consensus.cc:3037] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:09.921962 25463 raft_consensus.cc:504] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.922891 25463 leader_election.cc:288] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [CANDIDATE]: Term 1 election: Requested vote from peers 38bc27c659d546f59d46ab1a57b20dcf (127.19.246.195:38663), 40bbaee3931545058150a8169791abd2 (127.19.246.194:33339)
I20260810 03:19:09.924253 25385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8bc698ccbf40f4ac09f129a43af2a6" candidate_uuid: "7485e07147114b36a42378ef3c130dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38bc27c659d546f59d46ab1a57b20dcf"
I20260810 03:19:09.924104 25234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8bc698ccbf40f4ac09f129a43af2a6" candidate_uuid: "7485e07147114b36a42378ef3c130dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bbaee3931545058150a8169791abd2"
I20260810 03:19:09.924772 25385 raft_consensus.cc:3037] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:09.924811 25234 raft_consensus.cc:3037] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:09.927757 25385 raft_consensus.cc:2454] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7485e07147114b36a42378ef3c130dc8 in term 1.
I20260810 03:19:09.928844 25012 leader_election.cc:302] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38bc27c659d546f59d46ab1a57b20dcf, 7485e07147114b36a42378ef3c130dc8; no voters:
W20260810 03:19:09.928937 25435 tablet.cc:2369] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:19:09.929546 25463 raft_consensus.cc:2781] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:09.929914 25234 raft_consensus.cc:2454] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7485e07147114b36a42378ef3c130dc8 in term 1.
I20260810 03:19:09.930436 25463 raft_consensus.cc:686] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 7485e07147114b36a42378ef3c130dc8, State: Running, Role: LEADER
I20260810 03:19:09.931149 25463 consensus_queue.cc:237] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } }
I20260810 03:19:09.937054 24924 catalog_manager.cc:5495] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7485e07147114b36a42378ef3c130dc8 (127.19.246.193). New cstate: current_term: 1 leader_uuid: "7485e07147114b36a42378ef3c130dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7485e07147114b36a42378ef3c130dc8" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260810 03:19:09.990248 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:09.982686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.19.246.193:33021)
0810 03:19:09.982939 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:09.982944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:09.983103 (+ 159us) server_negotiation.cc:407] Connection header received
0810 03:19:09.983343 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:09.983424 (+ 81us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:09.983525 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:09.983683 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:09.984316 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.985249 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.986885 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:09.987658 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:09.987793 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:09.988753 (+ 960us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:09.989464 (+ 711us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:09.989608 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:09.989882 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 03:19:09.989970 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 03:19:10.000602 25234 raft_consensus.cc:1261] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 1 FOLLOWER]: Refusing update from remote peer 7485e07147114b36a42378ef3c130dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:10.000602 25385 raft_consensus.cc:1261] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 1 FOLLOWER]: Refusing update from remote peer 7485e07147114b36a42378ef3c130dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:10.001848 25468 consensus_queue.cc:1035] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38bc27c659d546f59d46ab1a57b20dcf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:10.002398 25463 consensus_queue.cc:1035] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40bbaee3931545058150a8169791abd2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:19:10.012624 25284 tablet.cc:2369] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:19:10.027222 25473 mvcc.cc:205] Tried to move back new op lower bound from 7316815667190075392 to 7316815666923483136. Current Snapshot: MvccSnapshot[applied={T|T < 7316815667190075392}]
I20260810 03:19:10.029279 25472 mvcc.cc:205] Tried to move back new op lower bound from 7316815667190075392 to 7316815666923483136. Current Snapshot: MvccSnapshot[applied={T|T < 7316815667190075392}]
I20260810 03:19:10.037874 25475 mvcc.cc:205] Tried to move back new op lower bound from 7316815667190075392 to 7316815666923483136. Current Snapshot: MvccSnapshot[applied={T|T < 7316815667190075392}]
I20260810 03:19:10.047832 24924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:10.048187 24924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:10.054093 24924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8da7da0b6a7d47b59d1e66a975776891: Sending DeleteTablet for 3 replicas of tablet fd8bc698ccbf40f4ac09f129a43af2a6
I20260810 03:19:10.055773 25362 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8bc698ccbf40f4ac09f129a43af2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:10 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:57932
I20260810 03:19:10.055778 25063 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8bc698ccbf40f4ac09f129a43af2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:10 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:43066
I20260810 03:19:10.055958 25211 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8bc698ccbf40f4ac09f129a43af2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:10 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:50312
I20260810 03:19:10.059841 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 24991
I20260810 03:19:10.061256 25484 tablet_replica.cc:330] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: stopping tablet replica
I20260810 03:19:10.061995 25484 raft_consensus.cc:2229] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:19:10.063125 25484 raft_consensus.cc:2258] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:10.066238 25484 ts_tablet_manager.cc:1905] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:10.076582 25483 tablet_replica.cc:330] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: stopping tablet replica
I20260810 03:19:10.077450 25483 raft_consensus.cc:2229] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:10.077688 25484 ts_tablet_manager.cc:1918] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:10.078147 25484 log.cc:1196] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/fd8bc698ccbf40f4ac09f129a43af2a6
I20260810 03:19:10.078362 25483 raft_consensus.cc:2258] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:10.079254 25484 ts_tablet_manager.cc:1939] T fd8bc698ccbf40f4ac09f129a43af2a6 P 7485e07147114b36a42378ef3c130dc8: Deleting consensus metadata
I20260810 03:19:10.081794 25483 ts_tablet_manager.cc:1905] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:10.082849 24912 catalog_manager.cc:4841] TS 7485e07147114b36a42378ef3c130dc8 (127.19.246.193:33021): tablet fd8bc698ccbf40f4ac09f129a43af2a6 (table test-table [id=c48bb10fa5614c05801dd5a85ba6fce5]) successfully deleted
Aug 10 03:19:10 dist-test-slave-btw8 krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:19:10.092425 25482 tablet_replica.cc:330] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: stopping tablet replica
I20260810 03:19:10.093755 25482 raft_consensus.cc:2229] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:10.094488 25482 raft_consensus.cc:2258] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:10.095077 25483 ts_tablet_manager.cc:1918] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:10.095332 25483 log.cc:1196] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/fd8bc698ccbf40f4ac09f129a43af2a6
I20260810 03:19:10.096611 25483 ts_tablet_manager.cc:1939] T fd8bc698ccbf40f4ac09f129a43af2a6 P 38bc27c659d546f59d46ab1a57b20dcf: Deleting consensus metadata
I20260810 03:19:10.097965 25482 ts_tablet_manager.cc:1905] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:10.098897 24912 catalog_manager.cc:4841] TS 38bc27c659d546f59d46ab1a57b20dcf (127.19.246.195:38663): tablet fd8bc698ccbf40f4ac09f129a43af2a6 (table test-table [id=c48bb10fa5614c05801dd5a85ba6fce5]) successfully deleted
I20260810 03:19:10.100287 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:10.061355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.19.246.193:33021)
0810 03:19:10.061546 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:10.061553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:10.064266 (+ 2713us) server_negotiation.cc:407] Connection header received
0810 03:19:10.064594 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:10.064603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:10.064721 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:10.064837 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:10.067633 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:10.069368 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:10.072522 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:10.073772 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:10.073934 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:10.088604 (+ 14670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:10.088631 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:10.088742 (+ 111us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:10.088816 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:10.094606 (+ 5790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:10.096362 (+ 1756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:10.096375 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:10.096380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:10.096801 (+ 421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:10.097492 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:10.097500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:10.097504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:10.098082 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:10.098442 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:10.099548 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0810 03:19:10.099844 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:19:10.101213 25130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:10.107936 25482 ts_tablet_manager.cc:1918] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:10.108217 25482 log.cc:1196] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/fd8bc698ccbf40f4ac09f129a43af2a6
I20260810 03:19:10.108856 25482 ts_tablet_manager.cc:1939] T fd8bc698ccbf40f4ac09f129a43af2a6 P 40bbaee3931545058150a8169791abd2: Deleting consensus metadata
I20260810 03:19:10.111255 24912 catalog_manager.cc:4841] TS 40bbaee3931545058150a8169791abd2 (127.19.246.194:33339): tablet fd8bc698ccbf40f4ac09f129a43af2a6 (table test-table [id=c48bb10fa5614c05801dd5a85ba6fce5]) successfully deleted
I20260810 03:19:10.212898 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 24991
I20260810 03:19:10.237775 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 25143
Aug 10 03:19:10 dist-test-slave-btw8 krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:19:10.261421 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:10.238826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.19.246.194:33339)
0810 03:19:10.239122 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:10.239128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:10.239157 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:19:10.239237 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:10.239242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:10.239324 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:10.239569 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:10.241322 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:10.242429 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:10.244274 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:10.244839 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:10.245101 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:10.251732 (+ 6631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:10.251774 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:10.251805 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:10.251883 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:10.256999 (+ 5116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:10.258503 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:10.258516 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:10.258519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:10.258753 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:10.259499 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:10.259504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:10.259507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:10.260012 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:10.260362 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:10.260846 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 03:19:10.261107 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260810 03:19:10.262454 25277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:10.363793 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 25143
I20260810 03:19:10.385655 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 25290
Aug 10 03:19:10 dist-test-slave-btw8 krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:19:10.404554 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:10.386572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.19.246.195:38663)
0810 03:19:10.386795 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:10.386804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:10.386906 (+ 102us) server_negotiation.cc:407] Connection header received
0810 03:19:10.387128 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:10.387134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:10.387247 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:10.387435 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:10.388842 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:10.389616 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:10.390925 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:10.391551 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:10.391688 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:10.397234 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:10.397257 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:10.397276 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:10.397332 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:10.400938 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:10.401857 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:10.401867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:10.401869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:10.402041 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:10.402589 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:10.402593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:10.402595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:10.403006 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:10.403260 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:10.403897 (+ 637us) server_negotiation.cc:299] Negotiation successful
0810 03:19:10.404197 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 03:19:10.405613 25428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:10.502776 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 25290
I20260810 03:19:10.526420 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 24888
I20260810 03:19:10.527782 24960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:10.643733 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 24888
2026-08-10T03:19:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5060 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T03:19:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:10Z Disabled control of system clock
I20260810 03:19:10.735818 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41223
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 03:19:11.121502 25493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:11.121865 25493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:11.121954 25493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:11.129979 25493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:11.130127 25493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:11.130165 25493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:11.130218 25493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:11.130255 25493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:11.141892 25493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:11.143460 25493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:11.146499 25493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:11.160506 25500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:11.160621 25499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:11.160570 25502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:11.161489 25493 server_base.cc:1034] running on GCE node
I20260810 03:19:11.162446 25493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:11.164263 25493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:11.165570 25493 hybrid_clock.cc:648] HybridClock initialized: now 1786331951165524 us; error 49 us; skew 500 ppm
I20260810 03:19:11.166110 25493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:19:11.169157 25493 webserver.cc:460] Webserver started at http://127.19.246.254:41483/ using document root <none> and password file <none>
I20260810 03:19:11.170061 25493 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:11.170228 25493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:11.170605 25493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:11.174192 25493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8"
format_stamp: "Formatted at 2026-08-10 03:19:11 on dist-test-slave-btw8"
server_key: "43dda446269edaf1f60dfc4addedf913"
server_key_iv: "a9897eb9e27ed5d9bd39db6221589a12"
server_key_version: "encryptionkey@0"
I20260810 03:19:11.175215 25493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8"
format_stamp: "Formatted at 2026-08-10 03:19:11 on dist-test-slave-btw8"
server_key: "43dda446269edaf1f60dfc4addedf913"
server_key_iv: "a9897eb9e27ed5d9bd39db6221589a12"
server_key_version: "encryptionkey@0"
I20260810 03:19:11.181666 25493 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:19:11.186911 25508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:11.188882 25493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:11.189165 25493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8"
format_stamp: "Formatted at 2026-08-10 03:19:11 on dist-test-slave-btw8"
server_key: "43dda446269edaf1f60dfc4addedf913"
server_key_iv: "a9897eb9e27ed5d9bd39db6221589a12"
server_key_version: "encryptionkey@0"
I20260810 03:19:11.189466 25493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:11.196658 25493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:11.197710 25493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:11.198076 25493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:11.222044 25493 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:41223
I20260810 03:19:11.222087 25565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:41223 every 8 connection(s)
I20260810 03:19:11.223999 25493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:11.227916 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 25493
I20260810 03:19:11.228154 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:11.228780 25566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:11.229022 20443 external_mini_cluster.cc:1428] Setting key 69f78e6c0cb4f0dbdc27d660f7c7d339
I20260810 03:19:11.238533 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:11.231282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.19.246.254:41223)
0810 03:19:11.232585 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:11.232598 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:11.232644 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:19:11.234241 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:11.234265 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:11.234760 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:11.235148 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:11.236665 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:11.236672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:11.236690 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:11.236711 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:11.236781 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:11.236804 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:11.237005 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:11.237174 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 03:19:11.237497 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":478,"threads_started":1}
I20260810 03:19:11.244294 25566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8: Bootstrap starting.
I20260810 03:19:11.247941 25566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:11.249089 25566 log.cc:824] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:11.252740 25566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8: No bootstrap required, opened a new log
I20260810 03:19:11.260737 25566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41223 } }
I20260810 03:19:11.261220 25566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:11.261337 25566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aae4ce2e7ba4a51bc2feda1e840cfd8, State: Initialized, Role: FOLLOWER
I20260810 03:19:11.261976 25566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41223 } }
I20260810 03:19:11.262241 25566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:11.262401 25566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:11.262607 25566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:11.265012 25566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41223 } }
I20260810 03:19:11.265632 25566 leader_election.cc:302] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aae4ce2e7ba4a51bc2feda1e840cfd8; no voters:
I20260810 03:19:11.266563 25566 leader_election.cc:288] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:11.266944 25571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:11.267979 25571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [term 1 LEADER]: Becoming Leader. State: Replica: 4aae4ce2e7ba4a51bc2feda1e840cfd8, State: Running, Role: LEADER
I20260810 03:19:11.268584 25571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41223 } }
I20260810 03:19:11.269671 25566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:11.272207 25572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41223 } } }
I20260810 03:19:11.272349 25573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aae4ce2e7ba4a51bc2feda1e840cfd8. Latest consensus state: current_term: 1 leader_uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aae4ce2e7ba4a51bc2feda1e840cfd8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41223 } } }
I20260810 03:19:11.272678 25572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:11.272682 25573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:11.274927 25580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:11.280316 25580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:11.290112 25580 catalog_manager.cc:1347] Generated new cluster ID: ca7807d2730947cd80da5bc78ea15679
I20260810 03:19:11.290266 25580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:11.317480 25580 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:11.318110 25580 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:11.329262 25580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8: Generated new TSK 0
I20260810 03:19:11.330431 25580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:19:11.344735 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41223
--builtin_ntp_servers=127.19.246.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 03:19:11.719744 25593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:11.720094 25593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:11.720275 25593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:11.728577 25593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:11.728724 25593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:11.728919 25593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:11.738955 25593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:11.740520 25593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:11.742663 25593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:11.760056 25598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:11.760003 25599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:11.760088 25601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:11.760887 25593 server_base.cc:1034] running on GCE node
I20260810 03:19:11.761793 25593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:11.763152 25593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:11.764430 25593 hybrid_clock.cc:648] HybridClock initialized: now 1786331951764347 us; error 75 us; skew 500 ppm
I20260810 03:19:11.764914 25593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:19:11.767971 25593 webserver.cc:460] Webserver started at http://127.19.246.193:44759/ using document root <none> and password file <none>
I20260810 03:19:11.768828 25593 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:11.768936 25593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:11.769364 25593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:11.773118 25593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "f127f064d451426e93ab6fa0270949a7"
format_stamp: "Formatted at 2026-08-10 03:19:11 on dist-test-slave-btw8"
server_key: "0e00d884911fce4320f82d4a75bd1fbe"
server_key_iv: "63b1231aa0e1fc662b6665ce200a0eb6"
server_key_version: "encryptionkey@0"
I20260810 03:19:11.774215 25593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "f127f064d451426e93ab6fa0270949a7"
format_stamp: "Formatted at 2026-08-10 03:19:11 on dist-test-slave-btw8"
server_key: "0e00d884911fce4320f82d4a75bd1fbe"
server_key_iv: "63b1231aa0e1fc662b6665ce200a0eb6"
server_key_version: "encryptionkey@0"
I20260810 03:19:11.781044 25593 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:19:11.785687 25607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:11.787858 25593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:11.788105 25593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "f127f064d451426e93ab6fa0270949a7"
format_stamp: "Formatted at 2026-08-10 03:19:11 on dist-test-slave-btw8"
server_key: "0e00d884911fce4320f82d4a75bd1fbe"
server_key_iv: "63b1231aa0e1fc662b6665ce200a0eb6"
server_key_version: "encryptionkey@0"
I20260810 03:19:11.788400 25593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:11.795789 25593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:11.797013 25593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:11.797271 25593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:11.798403 25593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:11.800096 25593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:11.800196 25593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:11.800279 25593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:11.800335 25593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:11.838238 25593 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:33869
I20260810 03:19:11.838385 25732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:33869 every 8 connection(s)
I20260810 03:19:11.839857 25593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:11.843906 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 25593
I20260810 03:19:11.844180 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:19:11.845042 20443 external_mini_cluster.cc:1428] Setting key 242af2aebb35e4690ad207605f973594
I20260810 03:19:11.848987 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41223
--builtin_ntp_servers=127.19.246.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 03:19:11.850384 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:11.842977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:54557 (local address 127.19.246.254:41223)
0810 03:19:11.843589 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:11.843596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:11.845274 (+ 1678us) server_negotiation.cc:407] Connection header received
0810 03:19:11.847427 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:11.847433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:11.847512 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:11.847639 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:11.849605 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:11.849610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:11.849613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:11.849630 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:11.849674 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:11.849687 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:11.849809 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:11.850073 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 03:19:11.850128 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":256,"threads_started":1}
I20260810 03:19:11.852128 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:11.843600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41223 (local address 127.19.246.193:54557)
0810 03:19:11.844952 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:11.845005 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:11.846994 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:11.848103 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:11.848119 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:11.848280 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:11.849150 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:19:11.849157 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:11.849233 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:19:11.849381 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:11.849815 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:11.849829 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:11.849842 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 03:19:11.850186 (+ 344us) client_negotiation.cc:240] Negotiation successful
0810 03:19:11.850525 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":385,"threads_started":1}
I20260810 03:19:11.854259 25733 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41223
I20260810 03:19:11.854645 25733 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:11.855312 25733 heartbeater.cc:511] Master 127.19.246.254:41223 requested a full tablet report, sending...
I20260810 03:19:11.857929 25527 ts_manager.cc:194] Registered new tserver with Master: f127f064d451426e93ab6fa0270949a7 (127.19.246.193:33869)
W20260810 03:19:12.220781 25738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:12.221180 25738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:12.221410 25738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:12.229409 25738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:12.229537 25738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:12.229760 25738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:12.239672 25738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:12.241128 25738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:12.243110 25738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:12.258394 25746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:12.258384 25743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:12.258441 25744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:12.260176 25738 server_base.cc:1034] running on GCE node
I20260810 03:19:12.261008 25738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:12.262152 25738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:12.263526 25738 hybrid_clock.cc:648] HybridClock initialized: now 1786331952263455 us; error 59 us; skew 500 ppm
I20260810 03:19:12.263923 25738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:19:12.266927 25738 webserver.cc:460] Webserver started at http://127.19.246.194:43545/ using document root <none> and password file <none>
I20260810 03:19:12.267863 25738 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:12.267968 25738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:12.268385 25738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:12.272246 25738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "8d459fc296f4438591c5e25796fb06c1"
format_stamp: "Formatted at 2026-08-10 03:19:12 on dist-test-slave-btw8"
server_key: "3ddc9d031ebdd51b034cfb0e7313e338"
server_key_iv: "9228b59bcc544a34a776244461832d26"
server_key_version: "encryptionkey@0"
I20260810 03:19:12.273351 25738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "8d459fc296f4438591c5e25796fb06c1"
format_stamp: "Formatted at 2026-08-10 03:19:12 on dist-test-slave-btw8"
server_key: "3ddc9d031ebdd51b034cfb0e7313e338"
server_key_iv: "9228b59bcc544a34a776244461832d26"
server_key_version: "encryptionkey@0"
I20260810 03:19:12.279901 25738 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:19:12.284526 25753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:12.286499 25738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:19:12.286772 25738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "8d459fc296f4438591c5e25796fb06c1"
format_stamp: "Formatted at 2026-08-10 03:19:12 on dist-test-slave-btw8"
server_key: "3ddc9d031ebdd51b034cfb0e7313e338"
server_key_iv: "9228b59bcc544a34a776244461832d26"
server_key_version: "encryptionkey@0"
I20260810 03:19:12.287029 25738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:12.295248 25738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:12.296491 25738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:12.296837 25738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:12.298116 25738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:12.300105 25738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:12.300216 25738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:12.300370 25738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:12.300459 25738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:12.340029 25738 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:33111
I20260810 03:19:12.340073 25878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:33111 every 8 connection(s)
I20260810 03:19:12.341634 25738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:12.349426 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 25738
I20260810 03:19:12.349694 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:19:12.350282 20443 external_mini_cluster.cc:1428] Setting key 17f6b7293497ff312966d1245939c912
I20260810 03:19:12.351621 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.344967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:34693 (local address 127.19.246.254:41223)
0810 03:19:12.345239 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:12.345245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:12.346824 (+ 1579us) server_negotiation.cc:407] Connection header received
0810 03:19:12.348830 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:12.348837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:12.348919 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:12.349054 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:12.350727 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:12.350733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:12.350738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:12.350769 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:12.350804 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:12.350818 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:12.350975 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:12.351179 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 03:19:12.351269 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 03:19:12.352905 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.345522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41223 (local address 127.19.246.194:34693)
0810 03:19:12.346636 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:12.346677 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:12.348491 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:12.349333 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:12.349343 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:12.349448 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:12.350311 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:19:12.350314 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:12.350394 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:19:12.350531 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:12.350922 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:12.350933 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:12.350944 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 03:19:12.351287 (+ 343us) client_negotiation.cc:240] Negotiation successful
0810 03:19:12.351642 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":335,"threads_started":1}
I20260810 03:19:12.353943 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41223
--builtin_ntp_servers=127.19.246.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 03:19:12.354872 25879 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41223
I20260810 03:19:12.355307 25879 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:12.356235 25879 heartbeater.cc:511] Master 127.19.246.254:41223 requested a full tablet report, sending...
I20260810 03:19:12.358222 25527 ts_manager.cc:194] Registered new tserver with Master: 8d459fc296f4438591c5e25796fb06c1 (127.19.246.194:33111)
W20260810 03:19:12.724547 25883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:12.724933 25883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:12.725113 25883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:12.732997 25883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:12.733186 25883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:12.733388 25883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:12.743278 25883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:12.744779 25883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:12.746695 25883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:12.762336 25889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:12.762439 25891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:12.762652 25888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:12.763178 25883 server_base.cc:1034] running on GCE node
I20260810 03:19:12.763908 25883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:12.764946 25883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:12.766247 25883 hybrid_clock.cc:648] HybridClock initialized: now 1786331952766170 us; error 41 us; skew 500 ppm
I20260810 03:19:12.766618 25883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:19:12.769598 25883 webserver.cc:460] Webserver started at http://127.19.246.195:36585/ using document root <none> and password file <none>
I20260810 03:19:12.770539 25883 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:12.770696 25883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:12.771108 25883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:12.774909 25883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "b85509d46c4a4173b87a02ead23f79ab"
format_stamp: "Formatted at 2026-08-10 03:19:12 on dist-test-slave-btw8"
server_key: "80bff006ed6584ba478da3b28fd4d9d4"
server_key_iv: "57ced153fa72f40b503fd2284bdc3f6c"
server_key_version: "encryptionkey@0"
I20260810 03:19:12.775949 25883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "b85509d46c4a4173b87a02ead23f79ab"
format_stamp: "Formatted at 2026-08-10 03:19:12 on dist-test-slave-btw8"
server_key: "80bff006ed6584ba478da3b28fd4d9d4"
server_key_iv: "57ced153fa72f40b503fd2284bdc3f6c"
server_key_version: "encryptionkey@0"
I20260810 03:19:12.782625 25883 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260810 03:19:12.787025 25898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:12.788956 25883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:12.789228 25883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "b85509d46c4a4173b87a02ead23f79ab"
format_stamp: "Formatted at 2026-08-10 03:19:12 on dist-test-slave-btw8"
server_key: "80bff006ed6584ba478da3b28fd4d9d4"
server_key_iv: "57ced153fa72f40b503fd2284bdc3f6c"
server_key_version: "encryptionkey@0"
I20260810 03:19:12.789520 25883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:12.796818 25883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:12.797850 25883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:12.798183 25883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:12.799268 25883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:12.801015 25883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:12.801164 25883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:12.801304 25883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:12.801395 25883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:12.841259 25883 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:34807
I20260810 03:19:12.841298 26026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:34807 every 8 connection(s)
I20260810 03:19:12.842732 25883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:12.843935 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 25883
I20260810 03:19:12.844156 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:19:12.844858 20443 external_mini_cluster.cc:1428] Setting key aa95da2cc74fae906da78998a5fef3fe
I20260810 03:19:12.853001 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.846169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:48031 (local address 127.19.246.254:41223)
0810 03:19:12.846580 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:12.846587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:12.848221 (+ 1634us) server_negotiation.cc:407] Connection header received
0810 03:19:12.850298 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:12.850304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:12.850391 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:12.850512 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:12.852136 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:12.852143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:12.852150 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:12.852186 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:12.852230 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:12.852245 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:12.852395 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:12.852606 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 03:19:12.852698 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260810 03:19:12.854122 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.846900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41223 (local address 127.19.246.195:48031)
0810 03:19:12.848060 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:12.848098 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:12.849992 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:12.850785 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:12.850795 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:12.850902 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:12.851749 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:19:12.851752 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:12.851796 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:19:12.851904 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:12.852359 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:12.852371 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:12.852379 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:19:12.852706 (+ 327us) client_negotiation.cc:240] Negotiation successful
0810 03:19:12.853032 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":453,"threads_started":1}
I20260810 03:19:12.855997 26027 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41223
I20260810 03:19:12.856474 26027 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:12.857264 26027 heartbeater.cc:511] Master 127.19.246.254:41223 requested a full tablet report, sending...
I20260810 03:19:12.859479 25527 ts_manager.cc:194] Registered new tserver with Master: b85509d46c4a4173b87a02ead23f79ab (127.19.246.195:34807)
I20260810 03:19:12.861814 25733 heartbeater.cc:503] Master 127.19.246.254:41223 was elected leader, sending a full tablet report...
I20260810 03:19:12.862361 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:19:12.871205 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.868103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.19.246.254:41223)
0810 03:19:12.868304 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:12.868310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:12.868844 (+ 534us) server_negotiation.cc:407] Connection header received
0810 03:19:12.869086 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:12.869091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:12.869151 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:12.869263 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:12.870455 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:12.870461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:12.870466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:12.870486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:12.870575 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:12.870596 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:12.870728 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:12.870828 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 03:19:12.870898 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:19:12.877233 25526 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:19:12.885630 25526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:12.922761 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.903028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.19.246.193:33869)
0810 03:19:12.918239 (+ 15211us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:12.918252 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:12.918298 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:19:12.918371 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:12.918378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:12.918627 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:12.918846 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:12.921479 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:12.921490 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:12.921508 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:12.921532 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:12.921610 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:12.921747 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:12.921970 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:12.922136 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 03:19:12.922294 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14979,"thread_start_us":16909,"threads_started":1}
I20260810 03:19:12.922857 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.902232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.0.0.1:60518)
0810 03:19:12.916103 (+ 13871us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:12.916138 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:12.916336 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:12.918881 (+ 2545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:12.918891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:12.919018 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:12.921060 (+ 2042us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:19:12.921063 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:12.921131 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:19:12.921233 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:12.921897 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:12.921914 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:12.921928 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 03:19:12.922145 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 03:19:12.922262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13584,"thread_start_us":12101,"threads_started":1}
I20260810 03:19:12.925949 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.902158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.19.246.194:33111)
0810 03:19:12.922478 (+ 20320us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:12.922489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:12.922529 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:19:12.922620 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:12.922629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:12.922925 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:12.923673 (+ 748us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:12.924982 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:12.924994 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:12.925012 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:12.925044 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:12.925113 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:12.925140 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:12.925357 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:12.925446 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 03:19:12.925560 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20088,"thread_start_us":19951,"threads_started":1}
I20260810 03:19:12.928021 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.901878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:33111 (local address 127.0.0.1:57238)
0810 03:19:12.913806 (+ 11928us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:12.913884 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:12.914140 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:12.923654 (+ 9514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:12.923660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:12.923687 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:12.924706 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:19:12.924709 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:12.924729 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:19:12.924775 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:12.925311 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:12.925317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:12.925321 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:19:12.925464 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 03:19:12.927512 (+ 2048us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11439,"thread_start_us":14586,"threads_started":1}
I20260810 03:19:12.930281 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.918940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:34807 (local address 127.0.0.1:37848)
0810 03:19:12.922849 (+ 3909us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:12.922871 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:12.923099 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:12.924763 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:12.924768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:12.924791 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:12.928668 (+ 3877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:19:12.928674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:12.928696 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:19:12.928763 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:12.929530 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:12.929538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:12.929546 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:19:12.929708 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 03:19:12.929783 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3757,"thread_start_us":4105,"threads_started":1}
I20260810 03:19:12.930552 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:12.916426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.19.246.195:34807)
0810 03:19:12.918860 (+ 2434us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:12.918881 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:12.922991 (+ 4110us) server_negotiation.cc:407] Connection header received
0810 03:19:12.923483 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:12.923501 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:12.923837 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:12.924617 (+ 780us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:12.929012 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:12.929024 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:12.929039 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:12.929080 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:12.929149 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:12.929182 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:12.929614 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:12.929694 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 03:19:12.929940 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2271,"thread_start_us":2077,"threads_started":1}
I20260810 03:19:12.928972 25660 tablet_service.cc:1467] Processing CreateTablet for tablet f9062d0fb5ad4e4c86c3ebf6364e8672 (DEFAULT_TABLE table=test-table [id=e33e115788c842c8aab6f30dbfd9905f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:12.932896 25660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9062d0fb5ad4e4c86c3ebf6364e8672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:12.933697 25807 tablet_service.cc:1467] Processing CreateTablet for tablet f9062d0fb5ad4e4c86c3ebf6364e8672 (DEFAULT_TABLE table=test-table [id=e33e115788c842c8aab6f30dbfd9905f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:12.935153 25807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9062d0fb5ad4e4c86c3ebf6364e8672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:12.936446 25953 tablet_service.cc:1467] Processing CreateTablet for tablet f9062d0fb5ad4e4c86c3ebf6364e8672 (DEFAULT_TABLE table=test-table [id=e33e115788c842c8aab6f30dbfd9905f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:12.938321 25953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9062d0fb5ad4e4c86c3ebf6364e8672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:12.948150 26047 tablet_bootstrap.cc:492] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Bootstrap starting.
I20260810 03:19:12.950604 26048 tablet_bootstrap.cc:492] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: Bootstrap starting.
I20260810 03:19:12.955564 26047 tablet_bootstrap.cc:654] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:12.956420 26048 tablet_bootstrap.cc:654] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:12.956506 26049 tablet_bootstrap.cc:492] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Bootstrap starting.
I20260810 03:19:12.958180 26047 log.cc:824] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:12.958251 26048 log.cc:824] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:12.960927 26049 tablet_bootstrap.cc:654] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:12.962337 26049 log.cc:824] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:12.962656 26048 tablet_bootstrap.cc:492] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: No bootstrap required, opened a new log
I20260810 03:19:12.962661 26047 tablet_bootstrap.cc:492] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: No bootstrap required, opened a new log
I20260810 03:19:12.963007 26048 ts_tablet_manager.cc:1397] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260810 03:19:12.963008 26047 ts_tablet_manager.cc:1397] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260810 03:19:12.965900 26049 tablet_bootstrap.cc:492] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: No bootstrap required, opened a new log
I20260810 03:19:12.966213 26049 ts_tablet_manager.cc:1397] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 03:19:12.971810 26047 raft_consensus.cc:348] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:12.973018 26047 raft_consensus.cc:374] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:12.973156 26047 raft_consensus.cc:729] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d459fc296f4438591c5e25796fb06c1, State: Initialized, Role: FOLLOWER
I20260810 03:19:12.974030 26047 consensus_queue.cc:260] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:12.974695 26048 raft_consensus.cc:348] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:12.975252 26048 raft_consensus.cc:374] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:12.975394 26048 raft_consensus.cc:729] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f127f064d451426e93ab6fa0270949a7, State: Initialized, Role: FOLLOWER
I20260810 03:19:12.976135 26048 consensus_queue.cc:260] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:12.978439 26048 ts_tablet_manager.cc:1428] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260810 03:19:12.978127 26049 raft_consensus.cc:348] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:12.978642 26049 raft_consensus.cc:374] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:12.978739 26049 raft_consensus.cc:729] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85509d46c4a4173b87a02ead23f79ab, State: Initialized, Role: FOLLOWER
I20260810 03:19:12.979565 25879 heartbeater.cc:503] Master 127.19.246.254:41223 was elected leader, sending a full tablet report...
I20260810 03:19:12.979925 26049 consensus_queue.cc:260] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:12.981480 26027 heartbeater.cc:503] Master 127.19.246.254:41223 was elected leader, sending a full tablet report...
I20260810 03:19:12.982843 26047 ts_tablet_manager.cc:1428] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Time spent starting tablet: real 0.020s user 0.016s sys 0.002s
I20260810 03:19:12.984265 26049 ts_tablet_manager.cc:1428] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Time spent starting tablet: real 0.018s user 0.011s sys 0.006s
I20260810 03:19:13.029646 26055 raft_consensus.cc:482] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:13.030016 26055 raft_consensus.cc:504] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:13.032106 26055 leader_election.cc:288] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d459fc296f4438591c5e25796fb06c1 (127.19.246.194:33111), f127f064d451426e93ab6fa0270949a7 (127.19.246.193:33869)
I20260810 03:19:13.035787 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.032464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:33111 (local address 127.19.246.195:34591)
0810 03:19:13.032916 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.032938 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:13.033069 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:13.033578 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:13.033584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:13.033618 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:13.034491 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:19:13.034494 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:13.034514 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:19:13.034571 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:13.035245 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:13.035250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:13.035254 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:19:13.035436 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 03:19:13.035499 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314}
I20260810 03:19:13.035948 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.032732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:34591 (local address 127.19.246.194:33111)
0810 03:19:13.032982 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:13.032990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:13.033018 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:19:13.033263 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:13.033271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:13.033366 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:13.033505 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:13.034862 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:13.034867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:13.034871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:13.034962 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:13.035014 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:13.035036 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:13.035214 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:13.035466 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 03:19:13.035526 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 03:19:13.036629 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.032678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:44617 (local address 127.19.246.193:33869)
0810 03:19:13.032918 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:13.032926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:13.034026 (+ 1100us) server_negotiation.cc:407] Connection header received
0810 03:19:13.034164 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:13.034172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:13.034250 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:13.034381 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:13.035696 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:13.035702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:13.035708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:13.035733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:13.035777 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:13.035796 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:13.035940 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:13.036255 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 03:19:13.036336 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 03:19:13.037259 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.032850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.19.246.195:44617)
0810 03:19:13.033855 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.033875 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:13.033994 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:13.034453 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:13.034458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:13.034482 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:13.035459 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:19:13.035463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:13.035480 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:19:13.035525 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:13.036024 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:13.036030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:13.036035 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:19:13.036882 (+ 847us) client_negotiation.cc:240] Negotiation successful
0810 03:19:13.036931 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":525,"threads_started":1}
I20260810 03:19:13.037240 25829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9062d0fb5ad4e4c86c3ebf6364e8672" candidate_uuid: "b85509d46c4a4173b87a02ead23f79ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d459fc296f4438591c5e25796fb06c1" is_pre_election: true
I20260810 03:19:13.037926 25829 raft_consensus.cc:2454] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85509d46c4a4173b87a02ead23f79ab in term 0.
I20260810 03:19:13.038414 25684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9062d0fb5ad4e4c86c3ebf6364e8672" candidate_uuid: "b85509d46c4a4173b87a02ead23f79ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f127f064d451426e93ab6fa0270949a7" is_pre_election: true
I20260810 03:19:13.038844 25902 leader_election.cc:302] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d459fc296f4438591c5e25796fb06c1, b85509d46c4a4173b87a02ead23f79ab; no voters:
I20260810 03:19:13.039113 25684 raft_consensus.cc:2454] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85509d46c4a4173b87a02ead23f79ab in term 0.
I20260810 03:19:13.039439 26055 raft_consensus.cc:2781] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:13.039714 26055 raft_consensus.cc:482] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:13.039889 26055 raft_consensus.cc:3037] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:13.042491 26055 raft_consensus.cc:504] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:13.043241 26055 leader_election.cc:288] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [CANDIDATE]: Term 1 election: Requested vote from peers 8d459fc296f4438591c5e25796fb06c1 (127.19.246.194:33111), f127f064d451426e93ab6fa0270949a7 (127.19.246.193:33869)
I20260810 03:19:13.043956 25829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9062d0fb5ad4e4c86c3ebf6364e8672" candidate_uuid: "b85509d46c4a4173b87a02ead23f79ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d459fc296f4438591c5e25796fb06c1"
I20260810 03:19:13.043999 25684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9062d0fb5ad4e4c86c3ebf6364e8672" candidate_uuid: "b85509d46c4a4173b87a02ead23f79ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f127f064d451426e93ab6fa0270949a7"
I20260810 03:19:13.044245 25684 raft_consensus.cc:3037] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:13.044246 25829 raft_consensus.cc:3037] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:13.048255 25684 raft_consensus.cc:2454] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85509d46c4a4173b87a02ead23f79ab in term 1.
I20260810 03:19:13.048255 25829 raft_consensus.cc:2454] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85509d46c4a4173b87a02ead23f79ab in term 1.
I20260810 03:19:13.048914 25902 leader_election.cc:302] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d459fc296f4438591c5e25796fb06c1, b85509d46c4a4173b87a02ead23f79ab; no voters:
I20260810 03:19:13.049314 26055 raft_consensus.cc:2781] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:13.049850 26055 raft_consensus.cc:686] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 1 LEADER]: Becoming Leader. State: Replica: b85509d46c4a4173b87a02ead23f79ab, State: Running, Role: LEADER
I20260810 03:19:13.050432 26055 consensus_queue.cc:237] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } }
I20260810 03:19:13.056499 25527 catalog_manager.cc:5495] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab reported cstate change: term changed from 0 to 1, leader changed from <none> to b85509d46c4a4173b87a02ead23f79ab (127.19.246.195). New cstate: current_term: 1 leader_uuid: "b85509d46c4a4173b87a02ead23f79ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85509d46c4a4173b87a02ead23f79ab" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 34807 } health_report { overall_health: HEALTHY } } }
I20260810 03:19:13.089416 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.086597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.19.246.195:34807)
0810 03:19:13.086919 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:13.086923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:13.086973 (+ 50us) server_negotiation.cc:407] Connection header received
0810 03:19:13.087267 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:13.087273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:13.087472 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:13.087601 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:13.088763 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:13.088768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:13.088772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:13.088792 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:13.088834 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:13.088848 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:13.088974 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:13.089119 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 03:19:13.089204 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
W20260810 03:19:13.092738 25734 tablet.cc:2369] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:19:13.095695 26028 tablet.cc:2369] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:19:13.095911 25880 tablet.cc:2369] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:19:13.099988 25684 raft_consensus.cc:1261] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 1 FOLLOWER]: Refusing update from remote peer b85509d46c4a4173b87a02ead23f79ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:13.100055 25829 raft_consensus.cc:1261] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 1 FOLLOWER]: Refusing update from remote peer b85509d46c4a4173b87a02ead23f79ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:13.101177 26057 consensus_queue.cc:1035] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f127f064d451426e93ab6fa0270949a7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:13.101691 26055 consensus_queue.cc:1035] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d459fc296f4438591c5e25796fb06c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:13.115710 26060 mvcc.cc:205] Tried to move back new op lower bound from 7316815679886254080 to 7316815679699615744. Current Snapshot: MvccSnapshot[applied={T|T < 7316815679886254080}]
I20260810 03:19:13.135674 25526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:13.136016 25526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:13.140422 25526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4aae4ce2e7ba4a51bc2feda1e840cfd8: Sending DeleteTablet for 3 replicas of tablet f9062d0fb5ad4e4c86c3ebf6364e8672
I20260810 03:19:13.142035 25807 tablet_service.cc:1514] Processing DeleteTablet for tablet f9062d0fb5ad4e4c86c3ebf6364e8672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:13 UTC) from {username='slave'} at 127.0.0.1:57238
I20260810 03:19:13.142073 25660 tablet_service.cc:1514] Processing DeleteTablet for tablet f9062d0fb5ad4e4c86c3ebf6364e8672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:13 UTC) from {username='slave'} at 127.0.0.1:60518
I20260810 03:19:13.142225 25953 tablet_service.cc:1514] Processing DeleteTablet for tablet f9062d0fb5ad4e4c86c3ebf6364e8672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:13 UTC) from {username='slave'} at 127.0.0.1:37848
I20260810 03:19:13.146457 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 25593
I20260810 03:19:13.155318 26073 tablet_replica.cc:330] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: stopping tablet replica
I20260810 03:19:13.156527 26072 tablet_replica.cc:330] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: stopping tablet replica
I20260810 03:19:13.157238 26073 raft_consensus.cc:2229] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:19:13.157012 26072 raft_consensus.cc:2229] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:13.157727 26071 tablet_replica.cc:330] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: stopping tablet replica
I20260810 03:19:13.158061 26072 raft_consensus.cc:2258] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:13.158191 26071 raft_consensus.cc:2229] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:13.158806 26071 raft_consensus.cc:2258] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:13.160132 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.148717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.19.246.193:33869)
0810 03:19:13.149014 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:13.149024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:13.156018 (+ 6994us) server_negotiation.cc:407] Connection header received
0810 03:19:13.156132 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:13.156141 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:13.156272 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:13.156388 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:13.159111 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:13.159118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:13.159122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:13.159148 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:13.159195 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:13.159217 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:13.159397 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:13.159715 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:19:13.159793 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 03:19:13.158841 26073 raft_consensus.cc:2258] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:13.161209 25728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:13.161597 26072 ts_tablet_manager.cc:1905] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:13.163295 26073 ts_tablet_manager.cc:1905] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:13.169576 26072 ts_tablet_manager.cc:1918] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:13.169809 26072 log.cc:1196] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/f9062d0fb5ad4e4c86c3ebf6364e8672
I20260810 03:19:13.171216 26072 ts_tablet_manager.cc:1939] T f9062d0fb5ad4e4c86c3ebf6364e8672 P 8d459fc296f4438591c5e25796fb06c1: Deleting consensus metadata
I20260810 03:19:13.172156 26073 ts_tablet_manager.cc:1918] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:13.172354 26073 log.cc:1196] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/f9062d0fb5ad4e4c86c3ebf6364e8672
I20260810 03:19:13.173002 26073 ts_tablet_manager.cc:1939] T f9062d0fb5ad4e4c86c3ebf6364e8672 P b85509d46c4a4173b87a02ead23f79ab: Deleting consensus metadata
I20260810 03:19:13.173655 25512 catalog_manager.cc:4841] TS 8d459fc296f4438591c5e25796fb06c1 (127.19.246.194:33111): tablet f9062d0fb5ad4e4c86c3ebf6364e8672 (table test-table [id=e33e115788c842c8aab6f30dbfd9905f]) successfully deleted
I20260810 03:19:13.176201 25513 catalog_manager.cc:4841] TS b85509d46c4a4173b87a02ead23f79ab (127.19.246.195:34807): tablet f9062d0fb5ad4e4c86c3ebf6364e8672 (table test-table [id=e33e115788c842c8aab6f30dbfd9905f]) successfully deleted
I20260810 03:19:13.162297 26071 ts_tablet_manager.cc:1905] T f9062d0fb5ad4e4c86c3ebf6364e8672 P f127f064d451426e93ab6fa0270949a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:13.281705 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 25593
W20260810 03:19:13.311829 25513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.246.193:33869 (error 108)
I20260810 03:19:13.312235 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 25738
I20260810 03:19:13.314239 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.313767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.0.0.1:60530)
0810 03:19:13.313953 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.314091 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 03:19:13.314601 25513 catalog_manager.cc:4584] TS f127f064d451426e93ab6fa0270949a7 (127.19.246.193:33869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9062d0fb5ad4e4c86c3ebf6364e8672: Network error: Client connection negotiation failed: client connection to 127.19.246.193:33869: connect: Connection refused (error 111)
I20260810 03:19:13.316430 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.313421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.19.246.194:33111)
0810 03:19:13.313622 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:13.313629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:13.313745 (+ 116us) server_negotiation.cc:407] Connection header received
0810 03:19:13.314054 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:13.314062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:13.314196 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:13.314316 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:13.315781 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:13.315786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:13.315789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:13.315808 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:13.315843 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:13.315860 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:13.315994 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:13.316140 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 03:19:13.316203 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:19:13.317323 25873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:13.368739 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.368061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.0.0.1:60540)
0810 03:19:13.368322 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.368438 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 03:19:13.370085 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.369631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.0.0.1:60556)
0810 03:19:13.369814 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.369907 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 03:19:13.412623 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 25738
I20260810 03:19:13.431950 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.431261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.0.0.1:60570)
0810 03:19:13.431597 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.431740 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260810 03:19:13.433504 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.432927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.0.0.1:60584)
0810 03:19:13.433151 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.433268 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 03:19:13.436100 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 25883
I20260810 03:19:13.439841 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.436923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.19.246.195:34807)
0810 03:19:13.437112 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:13.437116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:13.437216 (+ 100us) server_negotiation.cc:407] Connection header received
0810 03:19:13.437483 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:13.437493 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:13.437553 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:13.437651 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:13.438882 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:13.438888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:13.438891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:13.438912 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:13.438950 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:13.438965 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:13.439116 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:13.439325 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 03:19:13.439476 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:19:13.440912 26022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:13.515148 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.514498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.0.0.1:60596)
0810 03:19:13.514826 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.514939 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260810 03:19:13.516348 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:13.515909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33869 (local address 127.0.0.1:60606)
0810 03:19:13.516134 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:13.516212 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 03:19:13.567983 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 25883
I20260810 03:19:13.592063 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 25493
I20260810 03:19:13.593817 25560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:13.708844 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 25493
2026-08-10T03:19:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3060 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 03:19:13.758613 20443 mini_oidc.cc:152] Starting JWKS server
I20260810 03:19:13.763837 20443 webserver.cc:460] Webserver started at https://127.0.0.1:35665/ using document root <none> and password file <none>
I20260810 03:19:13.764298 20443 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:19:13.765142 20443 webserver.cc:460] Webserver started at http://127.0.0.1:34235/ using document root <none> and password file <none>
2026-08-10T03:19:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:13Z Disabled control of system clock
I20260810 03:19:13.807261 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:42965
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:42965
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35665/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/testchainca.cert with env {}
W20260810 03:19:14.186002 26086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:14.186420 26086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:14.186594 26086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:14.194343 26086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:14.194535 26086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:14.194639 26086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:14.194758 26086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:14.194856 26086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:19:14.194954 26086 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:19:14.195036 26086 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35665/jwks.json
I20260810 03:19:14.206616 26086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:42965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:42965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35665/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:14.208282 26086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:14.209816 26086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:14.223682 26095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:14.223771 26093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:14.223771 26092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:14.224990 26086 server_base.cc:1034] running on GCE node
I20260810 03:19:14.226023 26086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:14.227809 26086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:14.229125 26086 hybrid_clock.cc:648] HybridClock initialized: now 1786331954229017 us; error 72 us; skew 500 ppm
I20260810 03:19:14.229560 26086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:19:14.232908 26086 webserver.cc:460] Webserver started at http://127.19.246.254:38287/ using document root <none> and password file <none>
I20260810 03:19:14.234633 26086 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:14.235117 26086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:14.235649 26086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:14.239225 26086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "42e2148eaa474b6c8b429eade06009d8"
format_stamp: "Formatted at 2026-08-10 03:19:14 on dist-test-slave-btw8"
server_key: "ff1b4f7b15d213acac5a442ff46d5dfe"
server_key_iv: "fc887bc7e6ef6a222085f62917ed318b"
server_key_version: "encryptionkey@0"
I20260810 03:19:14.240334 26086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "42e2148eaa474b6c8b429eade06009d8"
format_stamp: "Formatted at 2026-08-10 03:19:14 on dist-test-slave-btw8"
server_key: "ff1b4f7b15d213acac5a442ff46d5dfe"
server_key_iv: "fc887bc7e6ef6a222085f62917ed318b"
server_key_version: "encryptionkey@0"
I20260810 03:19:14.246670 26086 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:19:14.251057 26102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:14.252945 26086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:14.253216 26086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "42e2148eaa474b6c8b429eade06009d8"
format_stamp: "Formatted at 2026-08-10 03:19:14 on dist-test-slave-btw8"
server_key: "ff1b4f7b15d213acac5a442ff46d5dfe"
server_key_iv: "fc887bc7e6ef6a222085f62917ed318b"
server_key_version: "encryptionkey@0"
I20260810 03:19:14.253504 26086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:14.277129 26086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:14.278267 26086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:14.278651 26086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:14.303341 26086 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:42965
I20260810 03:19:14.303467 26156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:42965 every 8 connection(s)
I20260810 03:19:14.305012 26086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:14.306874 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 26086
I20260810 03:19:14.307242 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:14.308077 20443 external_mini_cluster.cc:1428] Setting key d53165513ff8398686706e05de4777d4
I20260810 03:19:14.310225 26157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:14.328207 26157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8: Bootstrap starting.
I20260810 03:19:14.328336 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:14.310630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.19.246.254:42965)
0810 03:19:14.311986 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:14.312003 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:14.312068 (+ 65us) server_negotiation.cc:407] Connection header received
0810 03:19:14.314518 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:14.314557 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:14.315474 (+ 917us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:14.316363 (+ 889us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:14.317477 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:14.319875 (+ 2398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:14.321909 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:14.323028 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:14.323283 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:14.324882 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:14.324893 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:14.324916 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:14.325796 (+ 880us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:14.325903 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:14.325932 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:14.326065 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:14.326416 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 03:19:14.326814 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":400,"threads_started":1}
I20260810 03:19:14.332100 26157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:14.333433 26157 log.cc:824] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:14.334316 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:19:14.336946 26157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8: No bootstrap required, opened a new log
I20260810 03:19:14.344969 26157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2148eaa474b6c8b429eade06009d8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42965 } }
I20260810 03:19:14.345419 26157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:14.345563 26157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e2148eaa474b6c8b429eade06009d8, State: Initialized, Role: FOLLOWER
I20260810 03:19:14.346207 26157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2148eaa474b6c8b429eade06009d8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42965 } }
I20260810 03:19:14.346521 26157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:14.346688 26157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:14.346887 26157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:14.349602 26157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2148eaa474b6c8b429eade06009d8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42965 } }
I20260810 03:19:14.350196 26157 leader_election.cc:302] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42e2148eaa474b6c8b429eade06009d8; no voters:
I20260810 03:19:14.350871 26157 leader_election.cc:288] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:14.351092 26163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:14.351969 26163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [term 1 LEADER]: Becoming Leader. State: Replica: 42e2148eaa474b6c8b429eade06009d8, State: Running, Role: LEADER
I20260810 03:19:14.352720 26163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2148eaa474b6c8b429eade06009d8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42965 } }
I20260810 03:19:14.353216 26157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:14.356156 26164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42e2148eaa474b6c8b429eade06009d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2148eaa474b6c8b429eade06009d8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42965 } } }
I20260810 03:19:14.356633 26164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:14.357383 26165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e2148eaa474b6c8b429eade06009d8. Latest consensus state: current_term: 1 leader_uuid: "42e2148eaa474b6c8b429eade06009d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e2148eaa474b6c8b429eade06009d8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42965 } } }
I20260810 03:19:14.357766 26165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:14.359683 26172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:14.366182 26172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:14.376946 26172 catalog_manager.cc:1347] Generated new cluster ID: 1c1ee9bc1dfc4c9b8da39d9ec52a65f9
I20260810 03:19:14.377116 26172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:14.389544 26172 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:14.391711 26172 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:14.401080 26172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42e2148eaa474b6c8b429eade06009d8: Generated new TSK 0
I20260810 03:19:14.402338 26172 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 03:19:14.456647 26191 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 03:19:14.456961 26191 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:19:14.458248 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:14.427593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.19.246.254:42965)
0810 03:19:14.428012 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:14.428020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:14.428047 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:19:14.428454 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:14.428459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:14.428540 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:14.428728 (+ 188us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:19:14.429335 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:14.430100 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:14.431799 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:14.432369 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:14.432524 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:14.433198 (+ 674us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:19:14.457228 (+ 24030us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35665/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 03:19:14.457773 (+ 545us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44130: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35665/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":137}
W20260810 03:19:14.458400 26160 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44130: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35665/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 03:19:14.460090 26190 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:14.426984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42965 (local address 127.0.0.1:44130)
0810 03:19:14.427832 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:14.427864 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:14.428098 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:14.428755 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:14.428760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:14.428774 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:19:14.429165 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:14.429174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:14.430269 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:14.430275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:14.431634 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:14.431647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:14.432592 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:14.432602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:14.432758 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:14.432847 (+ 89us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:19:14.457615 (+ 24768us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35665/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 03:19:14.458002 (+ 387us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.246.254:42965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35665/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":425,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.246.254:42965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35665/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 03:19:14.462007 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 26086
I20260810 03:19:14.463557 26152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:14.564515 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 26086
2026-08-10T03:19:14Z chronyd exiting
I20260810 03:19:14.619201 20443 test_util.cc:175] -----------------------------------------------
I20260810 03:19:14.619431 20443 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786331905428759-20443-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (866 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 03:19:14.623732 20443 mini_oidc.cc:152] Starting JWKS server
I20260810 03:19:14.625891 20443 webserver.cc:460] Webserver started at https://127.0.0.1:39749/ using document root <none> and password file <none>
I20260810 03:19:14.626169 20443 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:19:14.626979 20443 webserver.cc:460] Webserver started at http://127.0.0.1:36399/ using document root <none> and password file <none>
2026-08-10T03:19:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:14Z Disabled control of system clock
I20260810 03:19:14.667274 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:46413
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:46413
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39749/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/testchainca.cert with env {}
W20260810 03:19:15.051412 26198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:15.051880 26198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:15.052119 26198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:15.060221 26198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:15.060424 26198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:15.060505 26198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:15.060595 26198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:15.060685 26198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:19:15.060776 26198 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:19:15.060856 26198 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39749/jwks.json
I20260810 03:19:15.071924 26198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:46413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:46413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39749/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:15.073441 26198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:15.074919 26198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:15.091078 26204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:15.090962 26203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:15.091037 26206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:15.092456 26198 server_base.cc:1034] running on GCE node
I20260810 03:19:15.093673 26198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:15.095785 26198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:15.097179 26198 hybrid_clock.cc:648] HybridClock initialized: now 1786331955097131 us; error 72 us; skew 500 ppm
I20260810 03:19:15.097690 26198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:19:15.106922 26198 webserver.cc:460] Webserver started at http://127.19.246.254:33809/ using document root <none> and password file <none>
I20260810 03:19:15.107931 26198 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:15.108110 26198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:15.108502 26198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:15.112301 26198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "c4a770cacd4a47699f7f4531d9851f46"
format_stamp: "Formatted at 2026-08-10 03:19:15 on dist-test-slave-btw8"
server_key: "2cf9a41db5055ee1c7be701c129bfdb8"
server_key_iv: "9dadb51fafd977a4c32c530ee5e0cb22"
server_key_version: "encryptionkey@0"
I20260810 03:19:15.113385 26198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "c4a770cacd4a47699f7f4531d9851f46"
format_stamp: "Formatted at 2026-08-10 03:19:15 on dist-test-slave-btw8"
server_key: "2cf9a41db5055ee1c7be701c129bfdb8"
server_key_iv: "9dadb51fafd977a4c32c530ee5e0cb22"
server_key_version: "encryptionkey@0"
I20260810 03:19:15.120491 26198 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260810 03:19:15.125389 26213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:15.127723 26198 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:19:15.127997 26198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "c4a770cacd4a47699f7f4531d9851f46"
format_stamp: "Formatted at 2026-08-10 03:19:15 on dist-test-slave-btw8"
server_key: "2cf9a41db5055ee1c7be701c129bfdb8"
server_key_iv: "9dadb51fafd977a4c32c530ee5e0cb22"
server_key_version: "encryptionkey@0"
I20260810 03:19:15.128751 26198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:15.143777 26198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:15.144806 26198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:15.145181 26198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:15.170248 26198 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:46413
I20260810 03:19:15.170262 26268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:46413 every 8 connection(s)
I20260810 03:19:15.172189 26198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:15.176990 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:15.177752 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 26198
I20260810 03:19:15.177970 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:15.178866 20443 external_mini_cluster.cc:1428] Setting key 06d38e379f2f74cbed945a3638b1d792
I20260810 03:19:15.195092 26269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46: Bootstrap starting.
I20260810 03:19:15.197816 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:15.181688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.19.246.254:46413)
0810 03:19:15.183042 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:15.183056 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:15.183112 (+ 56us) server_negotiation.cc:407] Connection header received
0810 03:19:15.185497 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:15.185532 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:15.186291 (+ 759us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:15.186779 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:15.187638 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:15.189077 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:15.191604 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:15.192508 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:15.192697 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:15.194395 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:15.194406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:15.194424 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:15.195062 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:15.195165 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:15.195323 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:15.195498 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:15.195835 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 03:19:15.196256 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":499,"threads_started":1}
I20260810 03:19:15.199774 26269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:15.201131 26269 log.cc:824] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:15.203827 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:19:15.205500 26269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46: No bootstrap required, opened a new log
I20260810 03:19:15.216797 26269 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a770cacd4a47699f7f4531d9851f46" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 46413 } }
I20260810 03:19:15.217258 26269 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:15.217334 26269 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a770cacd4a47699f7f4531d9851f46, State: Initialized, Role: FOLLOWER
I20260810 03:19:15.217995 26269 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a770cacd4a47699f7f4531d9851f46" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 46413 } }
I20260810 03:19:15.218251 26269 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:15.218361 26269 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:15.218501 26269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:15.220942 26269 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a770cacd4a47699f7f4531d9851f46" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 46413 } }
I20260810 03:19:15.221509 26269 leader_election.cc:302] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4a770cacd4a47699f7f4531d9851f46; no voters:
I20260810 03:19:15.222163 26269 leader_election.cc:288] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:15.222364 26275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:15.223290 26275 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [term 1 LEADER]: Becoming Leader. State: Replica: c4a770cacd4a47699f7f4531d9851f46, State: Running, Role: LEADER
I20260810 03:19:15.223922 26275 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a770cacd4a47699f7f4531d9851f46" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 46413 } }
I20260810 03:19:15.224736 26269 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:15.227376 26276 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4a770cacd4a47699f7f4531d9851f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a770cacd4a47699f7f4531d9851f46" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 46413 } } }
I20260810 03:19:15.227874 26276 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:15.228416 26277 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4a770cacd4a47699f7f4531d9851f46. Latest consensus state: current_term: 1 leader_uuid: "c4a770cacd4a47699f7f4531d9851f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a770cacd4a47699f7f4531d9851f46" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 46413 } } }
I20260810 03:19:15.228812 26277 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:15.229153 26283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:15.235802 26283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:15.248059 26283 catalog_manager.cc:1347] Generated new cluster ID: 71405defd1af4eb5945c11368245c7a1
I20260810 03:19:15.248268 26283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:15.272960 26283 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:15.274644 26283 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:15.284212 26283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4a770cacd4a47699f7f4531d9851f46: Generated new TSK 0
I20260810 03:19:15.285292 26283 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 03:19:15.326336 26301 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 03:19:15.326499 26301 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260810 03:19:15.328202 26300 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:15.303664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:46413 (local address 127.0.0.1:46482)
0810 03:19:15.304362 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:15.304386 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:15.304581 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:15.305198 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:15.305203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:15.305218 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:19:15.305566 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:15.305578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:15.306972 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:15.306979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:15.308970 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:15.308992 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:15.310142 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:15.310151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:15.310375 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:15.310430 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:19:15.327605 (+ 17175us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39749/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 03:19:15.327843 (+ 238us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.246.254:46413: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39749/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":375,"threads_started":1}
I20260810 03:19:15.328279 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:15.303810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.19.246.254:46413)
0810 03:19:15.304081 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:15.304088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:15.304541 (+ 453us) server_negotiation.cc:407] Connection header received
0810 03:19:15.304846 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:15.304851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:15.304947 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:15.305144 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:19:15.305786 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:15.306662 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:15.309231 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:15.309860 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:15.310003 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:15.310841 (+ 838us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:19:15.327261 (+ 16420us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39749/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 03:19:15.327913 (+ 652us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39749/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 03:19:15.328413 26272 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39749/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260810 03:19:15.330343 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 26198
I20260810 03:19:15.332419 26264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:15.436929 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 26198
2026-08-10T03:19:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (873 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 03:19:15.498090 20443 mini_oidc.cc:152] Starting JWKS server
I20260810 03:19:15.500499 20443 webserver.cc:460] Webserver started at https://127.0.0.1:44581/ using document root <none> and password file <none>
I20260810 03:19:15.500834 20443 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:19:15.501605 20443 webserver.cc:460] Webserver started at http://127.0.0.1:44409/ using document root <none> and password file <none>
2026-08-10T03:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:15Z Disabled control of system clock
I20260810 03:19:15.544049 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:33645
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:33645
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44581/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 03:19:15.945776 26308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:15.946125 26308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:15.946305 26308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:15.954496 26308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:15.954627 26308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:15.954674 26308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:15.954774 26308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:15.954860 26308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:19:15.954955 26308 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:19:15.955010 26308 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44581/jwks.json
I20260810 03:19:15.965193 26308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:33645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:33645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44581/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:15.966996 26308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:15.969630 26308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:15.984547 26316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:15.984656 26314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:15.984656 26313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:15.985714 26308 server_base.cc:1034] running on GCE node
I20260810 03:19:15.986652 26308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:15.988394 26308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:15.989665 26308 hybrid_clock.cc:648] HybridClock initialized: now 1786331955989592 us; error 68 us; skew 500 ppm
I20260810 03:19:15.990185 26308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:19:15.993314 26308 webserver.cc:460] Webserver started at http://127.19.246.254:39129/ using document root <none> and password file <none>
I20260810 03:19:15.994141 26308 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:15.994247 26308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:15.994611 26308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:15.998710 26308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "38176318cc154d4eb88498561f03d207"
format_stamp: "Formatted at 2026-08-10 03:19:15 on dist-test-slave-btw8"
server_key: "97c12a30727e4bb3092c711dbfe5971e"
server_key_iv: "a3ff18cf53e4e2831e51b0256c7d1b6b"
server_key_version: "encryptionkey@0"
I20260810 03:19:15.999835 26308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "38176318cc154d4eb88498561f03d207"
format_stamp: "Formatted at 2026-08-10 03:19:15 on dist-test-slave-btw8"
server_key: "97c12a30727e4bb3092c711dbfe5971e"
server_key_iv: "a3ff18cf53e4e2831e51b0256c7d1b6b"
server_key_version: "encryptionkey@0"
I20260810 03:19:16.006506 26308 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260810 03:19:16.011152 26322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:16.013111 26308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:16.013391 26308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "38176318cc154d4eb88498561f03d207"
format_stamp: "Formatted at 2026-08-10 03:19:15 on dist-test-slave-btw8"
server_key: "97c12a30727e4bb3092c711dbfe5971e"
server_key_iv: "a3ff18cf53e4e2831e51b0256c7d1b6b"
server_key_version: "encryptionkey@0"
I20260810 03:19:16.013664 26308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:16.045415 26308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:16.046610 26308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:16.047022 26308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:16.070547 26308 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:33645
I20260810 03:19:16.070569 26377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:33645 every 8 connection(s)
I20260810 03:19:16.072237 26308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:16.074707 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 26308
I20260810 03:19:16.074965 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:16.075739 20443 external_mini_cluster.cc:1428] Setting key bdeb001a5854619923065b3795cfbd34
I20260810 03:19:16.077412 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:16.094602 26378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207: Bootstrap starting.
I20260810 03:19:16.094726 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:16.078022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.19.246.254:33645)
0810 03:19:16.079291 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:16.079308 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:16.079397 (+ 89us) server_negotiation.cc:407] Connection header received
0810 03:19:16.081512 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:16.081549 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:16.082383 (+ 834us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:16.082980 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:16.083858 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:16.086184 (+ 2326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:16.088164 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:16.089332 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:16.089577 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:16.091222 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:16.091244 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:16.091269 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:16.092203 (+ 934us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:16.092316 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:16.092346 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:16.092478 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:16.092866 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 03:19:16.093284 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":392,"threads_started":1}
I20260810 03:19:16.098489 26378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:16.100255 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:19:16.100262 26378 log.cc:824] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:16.104586 26378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207: No bootstrap required, opened a new log
I20260810 03:19:16.113582 26378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38176318cc154d4eb88498561f03d207" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33645 } }
I20260810 03:19:16.114065 26378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:16.114142 26378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38176318cc154d4eb88498561f03d207, State: Initialized, Role: FOLLOWER
I20260810 03:19:16.114946 26378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38176318cc154d4eb88498561f03d207" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33645 } }
I20260810 03:19:16.115319 26378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:16.115535 26378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:16.115794 26378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:16.118495 26378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38176318cc154d4eb88498561f03d207" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33645 } }
I20260810 03:19:16.119190 26378 leader_election.cc:302] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38176318cc154d4eb88498561f03d207; no voters:
I20260810 03:19:16.119913 26378 leader_election.cc:288] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:16.120169 26383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:16.121908 26383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [term 1 LEADER]: Becoming Leader. State: Replica: 38176318cc154d4eb88498561f03d207, State: Running, Role: LEADER
I20260810 03:19:16.122676 26383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38176318cc154d4eb88498561f03d207" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33645 } }
I20260810 03:19:16.122948 26378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:16.126202 26384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38176318cc154d4eb88498561f03d207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38176318cc154d4eb88498561f03d207" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33645 } } }
I20260810 03:19:16.126134 26385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38176318cc154d4eb88498561f03d207. Latest consensus state: current_term: 1 leader_uuid: "38176318cc154d4eb88498561f03d207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38176318cc154d4eb88498561f03d207" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33645 } } }
I20260810 03:19:16.126699 26385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:16.126700 26384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:16.136957 26391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:16.142345 26391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:16.153151 26391 catalog_manager.cc:1347] Generated new cluster ID: c06ad9666d7d4f65bb4ce7584c197914
I20260810 03:19:16.153314 26391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:16.181746 26391 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:16.183336 26391 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:16.191665 26391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38176318cc154d4eb88498561f03d207: Generated new TSK 0
I20260810 03:19:16.192711 26391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:19:16.220413 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:16.210605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.19.246.254:33645)
0810 03:19:16.210841 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:16.210850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:16.211278 (+ 428us) server_negotiation.cc:407] Connection header received
0810 03:19:16.211590 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:16.211603 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:16.211706 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:16.211908 (+ 202us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:19:16.212501 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:16.213700 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:16.215302 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:16.216087 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:16.216247 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:16.217014 (+ 767us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:19:16.219663 (+ 2649us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 03:19:16.220041 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38816: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":94}
W20260810 03:19:16.220558 26381 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38816: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 03:19:16.222493 26409 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:16.210417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:33645 (local address 127.0.0.1:38816)
0810 03:19:16.211136 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:16.211156 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:16.211334 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:16.211968 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:16.211975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:16.211987 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:19:16.212330 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:16.212338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:16.213919 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:16.213924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:16.215117 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:16.215132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:16.216438 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:16.216445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:16.216614 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:16.216660 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:19:16.219935 (+ 3275us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 03:19:16.221996 (+ 2061us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.246.254:33645: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":1690,"thread_start_us":413,"threads_started":1}
I20260810 03:19:16.224440 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 26308
I20260810 03:19:16.226109 26373 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 03:19:16.236030 26410 webserver.cc:533] Webserver: sslize failed:
E20260810 03:19:16.236229 26410 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:19:16.329808 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 26308
2026-08-10T03:19:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (895 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T03:19:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:16Z Disabled control of system clock
I20260810 03:19:16.437798 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41265
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41265
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/testchainca.cert with env {}
W20260810 03:19:16.830565 26415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:16.830971 26415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:16.831072 26415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:16.840404 26415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:16.840557 26415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:16.840597 26415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:16.840653 26415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:16.840744 26415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:19:16.840853 26415 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:19:16.840911 26415 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 03:19:16.855950 26415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:16.857889 26415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:16.859938 26415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:16.877981 26422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:16.878191 26421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:16.878402 26424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:16.880519 26415 server_base.cc:1034] running on GCE node
I20260810 03:19:16.881852 26415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:16.884433 26415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:16.885783 26415 hybrid_clock.cc:648] HybridClock initialized: now 1786331956885626 us; error 133 us; skew 500 ppm
I20260810 03:19:16.886364 26415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:19:16.890787 26415 webserver.cc:460] Webserver started at http://127.19.246.254:39251/ using document root <none> and password file <none>
I20260810 03:19:16.892191 26415 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:16.892410 26415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:16.892865 26415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:16.898605 26415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "d0cca24b1f6b4f3b924ff43aebb06052"
format_stamp: "Formatted at 2026-08-10 03:19:16 on dist-test-slave-btw8"
server_key: "b98cf0f82703357bbc9c9dcdcc293a75"
server_key_iv: "6394ad47528418c3f96e702ce8583728"
server_key_version: "encryptionkey@0"
I20260810 03:19:16.900409 26415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "d0cca24b1f6b4f3b924ff43aebb06052"
format_stamp: "Formatted at 2026-08-10 03:19:16 on dist-test-slave-btw8"
server_key: "b98cf0f82703357bbc9c9dcdcc293a75"
server_key_iv: "6394ad47528418c3f96e702ce8583728"
server_key_version: "encryptionkey@0"
I20260810 03:19:16.910650 26415 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s
I20260810 03:19:16.916812 26431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:16.919032 26415 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260810 03:19:16.919498 26415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "d0cca24b1f6b4f3b924ff43aebb06052"
format_stamp: "Formatted at 2026-08-10 03:19:16 on dist-test-slave-btw8"
server_key: "b98cf0f82703357bbc9c9dcdcc293a75"
server_key_iv: "6394ad47528418c3f96e702ce8583728"
server_key_version: "encryptionkey@0"
I20260810 03:19:16.919824 26415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:16.944511 26415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:16.945829 26415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:16.946295 26415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:16.973248 26415 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:41265
I20260810 03:19:16.973271 26488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:41265 every 8 connection(s)
I20260810 03:19:16.975037 26415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:16.979724 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:16.984498 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 26415
I20260810 03:19:16.984798 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:16.985564 20443 external_mini_cluster.cc:1428] Setting key 93a6dad20d291f5196b6b7e7e603105f
I20260810 03:19:16.992820 26489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052: Bootstrap starting.
I20260810 03:19:16.996790 26489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:16.997997 26489 log.cc:824] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:17.004379 26489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052: No bootstrap required, opened a new log
I20260810 03:19:17.008983 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:16.987849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.19.246.254:41265)
0810 03:19:16.989574 (+ 1725us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:16.989591 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:16.989735 (+ 144us) server_negotiation.cc:407] Connection header received
0810 03:19:16.992067 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:16.992099 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:16.993216 (+ 1117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:16.993725 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:16.994588 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:16.996552 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:16.998292 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:16.999908 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:17.000197 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:17.002523 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:17.002533 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:17.002554 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:19:17.003925 (+ 1371us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:17.004037 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:19:17.004070 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:17.004194 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:17.004773 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 03:19:17.005891 (+ 1118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":542,"threads_started":1}
I20260810 03:19:17.016369 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:19:17.018270 26489 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cca24b1f6b4f3b924ff43aebb06052" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41265 } }
I20260810 03:19:17.018854 26489 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:17.018968 26489 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cca24b1f6b4f3b924ff43aebb06052, State: Initialized, Role: FOLLOWER
I20260810 03:19:17.019743 26489 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cca24b1f6b4f3b924ff43aebb06052" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41265 } }
I20260810 03:19:17.020032 26489 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:17.020552 26489 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:17.020956 26489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:17.024735 26489 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cca24b1f6b4f3b924ff43aebb06052" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41265 } }
I20260810 03:19:17.026081 26489 leader_election.cc:302] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0cca24b1f6b4f3b924ff43aebb06052; no voters:
I20260810 03:19:17.027026 26489 leader_election.cc:288] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:17.027206 26495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:17.029379 26495 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [term 1 LEADER]: Becoming Leader. State: Replica: d0cca24b1f6b4f3b924ff43aebb06052, State: Running, Role: LEADER
I20260810 03:19:17.030095 26495 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cca24b1f6b4f3b924ff43aebb06052" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41265 } }
I20260810 03:19:17.030643 26489 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:17.033586 26497 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0cca24b1f6b4f3b924ff43aebb06052. Latest consensus state: current_term: 1 leader_uuid: "d0cca24b1f6b4f3b924ff43aebb06052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cca24b1f6b4f3b924ff43aebb06052" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41265 } } }
I20260810 03:19:17.034274 26497 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:17.034175 26496 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0cca24b1f6b4f3b924ff43aebb06052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cca24b1f6b4f3b924ff43aebb06052" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41265 } } }
I20260810 03:19:17.034619 26496 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:17.035640 26503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:17.044801 26503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:17.058455 26503 catalog_manager.cc:1347] Generated new cluster ID: 997d2dfc3935446eb40586c3c0055c57
I20260810 03:19:17.058640 26503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:17.089412 26503 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:17.092010 26503 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:17.107219 26503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0cca24b1f6b4f3b924ff43aebb06052: Generated new TSK 0
I20260810 03:19:17.108620 26503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:19:17.222162 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:17.136019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.19.246.254:41265)
0810 03:19:17.136271 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:17.136276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:17.137515 (+ 1239us) server_negotiation.cc:407] Connection header received
0810 03:19:17.137917 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:17.137924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:17.138064 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:17.138552 (+ 488us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:19:17.139164 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:17.140844 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:17.143122 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:17.144022 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:17.144232 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:17.144871 (+ 639us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:19:17.221221 (+ 76350us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 03:19:17.221718 (+ 497us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52186: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":94}
W20260810 03:19:17.222348 26492 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52186: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 03:19:17.222656 26521 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:17.136412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41265 (local address 127.0.0.1:52186)
0810 03:19:17.137267 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:17.137286 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:17.137475 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:17.138597 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:17.138608 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:17.138621 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:19:17.138936 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:17.138945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:17.141344 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:17.141351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:17.142841 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:17.142864 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:17.144317 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:17.144323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:17.144475 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:17.144519 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:19:17.221687 (+ 77168us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 03:19:17.222166 (+ 479us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.246.254:41265: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":462,"threads_started":1}
I20260810 03:19:17.224680 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 26415
I20260810 03:19:17.226263 26484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:17.343767 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 26415
2026-08-10T03:19:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (993 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (502 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1724 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:19:19 dist-test-slave-btw8 krb5kdc[26538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:19:19 dist-test-slave-btw8 krb5kdc[26538](info): set up 2 sockets
Aug 10 03:19:19 dist-test-slave-btw8 krb5kdc[26538](info): commencing operation
krb5kdc: starting...
W20260810 03:19:20.909427 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.001s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:19:21 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:19:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:19:21.237572 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36837
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:36837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:21.701578 26553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:21.702157 26553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:21.702343 26553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:21.712069 26553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:21.712410 26553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:21.712525 26553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:21.712638 26553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:21.712806 26553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:21.725477 26553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:36837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:21.727289 26553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:21.728929 26553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:21.745280 26559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:21.745247 26562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:21.745182 26560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:21.747466 26553 server_base.cc:1034] running on GCE node
I20260810 03:19:21.748878 26553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:21.751832 26553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:21.753147 26553 hybrid_clock.cc:648] HybridClock initialized: now 1786331961753059 us; error 108 us; skew 500 ppm
I20260810 03:19:21.753750 26553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:21 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331961, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:21.763758 26553 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:19:21.765873 26553 webserver.cc:460] Webserver started at http://127.19.246.254:45197/ using document root <none> and password file <none>
I20260810 03:19:21.766806 26553 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:21.767010 26553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:21.767503 26553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:21.771540 26553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "616bcd42bd704dfb89aafdfc2e9c88d1"
format_stamp: "Formatted at 2026-08-10 03:19:21 on dist-test-slave-btw8"
server_key: "03bc4436eaae8f8297b21af3cfa53710"
server_key_iv: "e7c9e5dbd855539d9a374a30aacb9c66"
server_key_version: "encryptionkey@0"
I20260810 03:19:21.772749 26553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "616bcd42bd704dfb89aafdfc2e9c88d1"
format_stamp: "Formatted at 2026-08-10 03:19:21 on dist-test-slave-btw8"
server_key: "03bc4436eaae8f8297b21af3cfa53710"
server_key_iv: "e7c9e5dbd855539d9a374a30aacb9c66"
server_key_version: "encryptionkey@0"
I20260810 03:19:21.781060 26553 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260810 03:19:21.787730 26570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:21.789934 26553 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260810 03:19:21.790382 26553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "616bcd42bd704dfb89aafdfc2e9c88d1"
format_stamp: "Formatted at 2026-08-10 03:19:21 on dist-test-slave-btw8"
server_key: "03bc4436eaae8f8297b21af3cfa53710"
server_key_iv: "e7c9e5dbd855539d9a374a30aacb9c66"
server_key_version: "encryptionkey@0"
I20260810 03:19:21.790656 26553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:21.828806 26553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:21.835248 26553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:21.835678 26553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:21.872331 26553 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:36837
I20260810 03:19:21.872395 26625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:36837 every 8 connection(s)
I20260810 03:19:21.874961 26553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:21.877375 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 26553
I20260810 03:19:21.877821 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:21.879062 20443 external_mini_cluster.cc:1428] Setting key 29966e1cc084a5a8bd9830d9e58f1d3a
I20260810 03:19:21.884028 26626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:19:21 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:21.902884 26626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1: Bootstrap starting.
I20260810 03:19:21.911520 26626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:21.913529 26626 log.cc:824] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:21.919615 26626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1: No bootstrap required, opened a new log
I20260810 03:19:21.923586 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:21.883068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.19.246.254:36837)
0810 03:19:21.884216 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:21.884241 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:21.884319 (+ 78us) server_negotiation.cc:407] Connection header received
0810 03:19:21.887047 (+ 2728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:21.887079 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:21.887700 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:21.888291 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:21.890354 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:21.892282 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:21.893865 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:21.895140 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:21.895539 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:21.906236 (+ 10697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:21.906273 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:21.906521 (+ 248us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:21.906655 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:21.913060 (+ 6405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:21.914744 (+ 1684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:21.914759 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:21.914772 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:21.915114 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:21.915876 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:21.915881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:21.915884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:21.916849 (+ 965us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:21.917148 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:21.919109 (+ 1961us) server_negotiation.cc:299] Negotiation successful
0810 03:19:21.920188 (+ 1079us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":528,"threads_started":1}
I20260810 03:19:21.934587 26626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616bcd42bd704dfb89aafdfc2e9c88d1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36837 } }
I20260810 03:19:21.935937 26626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:21.936167 26626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616bcd42bd704dfb89aafdfc2e9c88d1, State: Initialized, Role: FOLLOWER
I20260810 03:19:21.937865 26626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616bcd42bd704dfb89aafdfc2e9c88d1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36837 } }
I20260810 03:19:21.938365 26626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:21.938512 26626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:21.938877 26626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:21.943552 26626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616bcd42bd704dfb89aafdfc2e9c88d1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36837 } }
I20260810 03:19:21.944286 26626 leader_election.cc:302] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 616bcd42bd704dfb89aafdfc2e9c88d1; no voters:
I20260810 03:19:21.945042 26626 leader_election.cc:288] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:21.945340 26632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:21.946449 26632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [term 1 LEADER]: Becoming Leader. State: Replica: 616bcd42bd704dfb89aafdfc2e9c88d1, State: Running, Role: LEADER
I20260810 03:19:21.947280 26632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616bcd42bd704dfb89aafdfc2e9c88d1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36837 } }
I20260810 03:19:21.949155 26626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:21.951218 26634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 616bcd42bd704dfb89aafdfc2e9c88d1. Latest consensus state: current_term: 1 leader_uuid: "616bcd42bd704dfb89aafdfc2e9c88d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616bcd42bd704dfb89aafdfc2e9c88d1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36837 } } }
I20260810 03:19:21.951932 26634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:21.952667 26633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "616bcd42bd704dfb89aafdfc2e9c88d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616bcd42bd704dfb89aafdfc2e9c88d1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36837 } } }
I20260810 03:19:21.953037 26633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:21.953217 26640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:21.962050 26640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:21.974715 26640 catalog_manager.cc:1347] Generated new cluster ID: 9e75e2c748c841468ed1aaa75f5df8b5
I20260810 03:19:21.974872 26640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:21.987550 26640 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:21.989846 26640 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:22.002174 26640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 616bcd42bd704dfb89aafdfc2e9c88d1: Generated new TSK 0
I20260810 03:19:22.004184 26640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:19:22.104214 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36837
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:22.561158 26654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:22.562052 26654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:22.562292 26654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:22.574470 26654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:22.574738 26654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:22.574962 26654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:22.586896 26654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:22.588565 26654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:22.590234 26654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:22.608778 26662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:22.608872 26661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:22.608881 26664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:22.609651 26654 server_base.cc:1034] running on GCE node
I20260810 03:19:22.610472 26654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:22.611814 26654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:22.613102 26654 hybrid_clock.cc:648] HybridClock initialized: now 1786331962613039 us; error 56 us; skew 500 ppm
I20260810 03:19:22.613554 26654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:22 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:22.618216 26654 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:19:22.620440 26654 webserver.cc:460] Webserver started at http://127.19.246.193:43965/ using document root <none> and password file <none>
I20260810 03:19:22.621358 26654 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:22.621549 26654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:22.621932 26654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:22.625499 26654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "2a8a2346b377435b93accdb0662bd083"
format_stamp: "Formatted at 2026-08-10 03:19:22 on dist-test-slave-btw8"
server_key: "ab5e60b6de05aaac0038264831522004"
server_key_iv: "4b98021edb1ee907ddf751e719b1064f"
server_key_version: "encryptionkey@0"
I20260810 03:19:22.626487 26654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "2a8a2346b377435b93accdb0662bd083"
format_stamp: "Formatted at 2026-08-10 03:19:22 on dist-test-slave-btw8"
server_key: "ab5e60b6de05aaac0038264831522004"
server_key_iv: "4b98021edb1ee907ddf751e719b1064f"
server_key_version: "encryptionkey@0"
I20260810 03:19:22.633143 26654 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 03:19:22.638648 26671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:22.641449 26654 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260810 03:19:22.641754 26654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "2a8a2346b377435b93accdb0662bd083"
format_stamp: "Formatted at 2026-08-10 03:19:22 on dist-test-slave-btw8"
server_key: "ab5e60b6de05aaac0038264831522004"
server_key_iv: "4b98021edb1ee907ddf751e719b1064f"
server_key_version: "encryptionkey@0"
I20260810 03:19:22.642037 26654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:22.671025 26654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:22.677871 26654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:22.678349 26654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:22.679713 26654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:22.681660 26654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:22.681813 26654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:22.682027 26654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:22.682123 26654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:22.732283 26654 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:44789
I20260810 03:19:22.732304 26798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:44789 every 8 connection(s)
I20260810 03:19:22.734776 26654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:22.743780 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 26654
I20260810 03:19:22.744067 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:19:22.744936 20443 external_mini_cluster.cc:1428] Setting key 81744a9cf42f80862a120c621b780a2e
Aug 10 03:19:22 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:22.765134 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:22.739837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:59599 (local address 127.19.246.254:36837)
0810 03:19:22.740784 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:22.740790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:22.741913 (+ 1123us) server_negotiation.cc:407] Connection header received
0810 03:19:22.744321 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:22.744328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:22.744441 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:22.744574 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:22.746667 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:22.747979 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:22.750118 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:22.750669 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:22.750827 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:22.755842 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:22.755853 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:22.755861 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:22.755921 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:22.760041 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:22.761801 (+ 1760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:22.761807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:22.761809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:22.761989 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:22.762689 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:22.762694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:22.762697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:22.763198 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:22.763592 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:22.764567 (+ 975us) server_negotiation.cc:299] Negotiation successful
0810 03:19:22.764813 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":586,"threads_started":1}
I20260810 03:19:22.766901 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:22.740356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36837 (local address 127.19.246.193:59599)
0810 03:19:22.741681 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:22.741729 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:22.743830 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:22.744889 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:22.744899 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:22.745586 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:22.746367 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:22.746397 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:22.748259 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:22.748265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:22.749914 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:22.749932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:22.750948 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:22.750955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:22.751252 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:22.752223 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:22.752251 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:22.755497 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:22.760287 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:22.760301 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:22.760326 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:22.761406 (+ 1080us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:22.762158 (+ 752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:22.762166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:22.762172 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:22.762508 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:22.763427 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:22.763440 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:22.764020 (+ 580us) client_negotiation.cc:769] Sending connection context
0810 03:19:22.764414 (+ 394us) client_negotiation.cc:240] Negotiation successful
0810 03:19:22.765144 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":495,"threads_started":1}
I20260810 03:19:22.769006 26799 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36837
I20260810 03:19:22.769474 26799 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:22.770402 26799 heartbeater.cc:511] Master 127.19.246.254:36837 requested a full tablet report, sending...
I20260810 03:19:22.773453 26587 ts_manager.cc:194] Registered new tserver with Master: 2a8a2346b377435b93accdb0662bd083 (127.19.246.193:44789)
I20260810 03:19:22.775153 26587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:59599
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:19:22.824563 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36837
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:23.247938 26806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:23.248353 26806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:23.248543 26806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:23.257961 26806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:23.258314 26806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:23.259008 26806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:23.272897 26806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:23.274544 26806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:23.276146 26806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:23.292644 26811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:23.292724 26812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:23.292680 26814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:23.294117 26806 server_base.cc:1034] running on GCE node
I20260810 03:19:23.295004 26806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:23.296499 26806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:23.298125 26806 hybrid_clock.cc:648] HybridClock initialized: now 1786331963298051 us; error 73 us; skew 500 ppm
I20260810 03:19:23.298717 26806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:23 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331963, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:23.303566 26806 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:19:23.306725 26806 webserver.cc:460] Webserver started at http://127.19.246.194:35881/ using document root <none> and password file <none>
I20260810 03:19:23.307845 26806 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:23.308040 26806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:23.308482 26806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:23.314952 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "ac37bd8165174b5d827b7647c304ad80"
format_stamp: "Formatted at 2026-08-10 03:19:23 on dist-test-slave-btw8"
server_key: "c853c0b355823fb72f3b3e157e034051"
server_key_iv: "aaa17ba67326b14b6fc232f603be0753"
server_key_version: "encryptionkey@0"
I20260810 03:19:23.316494 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "ac37bd8165174b5d827b7647c304ad80"
format_stamp: "Formatted at 2026-08-10 03:19:23 on dist-test-slave-btw8"
server_key: "c853c0b355823fb72f3b3e157e034051"
server_key_iv: "aaa17ba67326b14b6fc232f603be0753"
server_key_version: "encryptionkey@0"
I20260810 03:19:23.325734 26806 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260810 03:19:23.330690 26823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:23.332748 26806 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:19:23.333006 26806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "ac37bd8165174b5d827b7647c304ad80"
format_stamp: "Formatted at 2026-08-10 03:19:23 on dist-test-slave-btw8"
server_key: "c853c0b355823fb72f3b3e157e034051"
server_key_iv: "aaa17ba67326b14b6fc232f603be0753"
server_key_version: "encryptionkey@0"
I20260810 03:19:23.333276 26806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:23.351657 26806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:23.357537 26806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:23.357920 26806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:23.359251 26806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:23.361331 26806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:23.361465 26806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:23.361660 26806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:23.361747 26806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:23.426430 26806 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:40179
I20260810 03:19:23.426460 26945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:40179 every 8 connection(s)
I20260810 03:19:23.429447 26806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:23.430634 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 26806
I20260810 03:19:23.430892 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:19:23.431702 20443 external_mini_cluster.cc:1428] Setting key e279ea997fa8159d0511143f54296a7b
Aug 10 03:19:23 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331963, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:23.471845 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:23.434762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:34255 (local address 127.19.246.254:36837)
0810 03:19:23.438674 (+ 3912us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:23.438717 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:23.438901 (+ 184us) server_negotiation.cc:407] Connection header received
0810 03:19:23.440340 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:23.440351 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:23.440476 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:23.440655 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:23.445672 (+ 5017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:23.447250 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:23.450951 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:23.451820 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:23.452032 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:23.458362 (+ 6330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:23.458375 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:23.458384 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:23.458453 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:23.463618 (+ 5165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:23.465421 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:23.465432 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:23.465439 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:23.465713 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:23.468039 (+ 2326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:23.468047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:23.468051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:23.468957 (+ 906us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:23.469641 (+ 684us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:23.470970 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0810 03:19:23.471381 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3231,"thread_start_us":2839,"threads_started":1}
I20260810 03:19:23.472332 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:23.434769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36837 (local address 127.19.246.194:34255)
0810 03:19:23.436363 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:23.436452 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:23.439922 (+ 3470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:23.441469 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:23.441492 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:23.443444 (+ 1952us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:23.445113 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:23.445156 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:23.447952 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:23.447961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:23.450543 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:23.450640 (+ 97us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:23.452343 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:23.452355 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:23.452806 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:23.454261 (+ 1455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:23.454293 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:23.457926 (+ 3633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:23.464083 (+ 6157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:23.464098 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:23.464124 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:23.465133 (+ 1009us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:23.466058 (+ 925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:23.466065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:23.466069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:23.467750 (+ 1681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:23.469555 (+ 1805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:23.469567 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:23.470128 (+ 561us) client_negotiation.cc:769] Sending connection context
0810 03:19:23.470417 (+ 289us) client_negotiation.cc:240] Negotiation successful
0810 03:19:23.470919 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":554,"threads_started":1}
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
I20260810 03:19:23.475199 26946 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36837
I20260810 03:19:23.476146 26946 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:23.477025 26946 heartbeater.cc:511] Master 127.19.246.254:36837 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
I20260810 03:19:23.480847 26587 ts_manager.cc:194] Registered new tserver with Master: ac37bd8165174b5d827b7647c304ad80 (127.19.246.194:40179)
I20260810 03:19:23.482435 26587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:34255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:19:23.534713 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36837
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:23.779417 26799 heartbeater.cc:503] Master 127.19.246.254:36837 was elected leader, sending a full tablet report...
W20260810 03:19:23.984244 26952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:23.984602 26952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:23.984726 26952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:23.992897 26952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:23.993050 26952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:23.993172 26952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:24.005950 26952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:24.007453 26952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:24.008985 26952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:24.028861 26958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:24.028962 26960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:24.028893 26957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:24.030113 26952 server_base.cc:1034] running on GCE node
I20260810 03:19:24.030952 26952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:24.032637 26952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:24.036336 26952 hybrid_clock.cc:648] HybridClock initialized: now 1786331964036105 us; error 253 us; skew 500 ppm
I20260810 03:19:24.036948 26952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:24.041448 26952 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:19:24.043257 26952 webserver.cc:460] Webserver started at http://127.19.246.195:42675/ using document root <none> and password file <none>
I20260810 03:19:24.044162 26952 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:24.044289 26952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:24.044587 26952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:24.048372 26952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "a1ddf38af28749948c5b32e444bc8964"
format_stamp: "Formatted at 2026-08-10 03:19:24 on dist-test-slave-btw8"
server_key: "498074e572f71fbedc86a5f578c893ce"
server_key_iv: "06a403c309ab13f97382bc9729bb2714"
server_key_version: "encryptionkey@0"
I20260810 03:19:24.049996 26952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "a1ddf38af28749948c5b32e444bc8964"
format_stamp: "Formatted at 2026-08-10 03:19:24 on dist-test-slave-btw8"
server_key: "498074e572f71fbedc86a5f578c893ce"
server_key_iv: "06a403c309ab13f97382bc9729bb2714"
server_key_version: "encryptionkey@0"
I20260810 03:19:24.060362 26952 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260810 03:19:24.066103 26967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:24.068234 26952 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260810 03:19:24.068463 26952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "a1ddf38af28749948c5b32e444bc8964"
format_stamp: "Formatted at 2026-08-10 03:19:24 on dist-test-slave-btw8"
server_key: "498074e572f71fbedc86a5f578c893ce"
server_key_iv: "06a403c309ab13f97382bc9729bb2714"
server_key_version: "encryptionkey@0"
I20260810 03:19:24.068717 26952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:24.095129 26952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:24.102092 26952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:24.102401 26952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:24.103664 26952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:24.105381 26952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:24.105468 26952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:24.105584 26952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:24.105625 26952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:24.149842 26952 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:33571
I20260810 03:19:24.149998 27091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:33571 every 8 connection(s)
I20260810 03:19:24.151491 26952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:24.153102 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 26952
I20260810 03:19:24.153314 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:19:24.153839 20443 external_mini_cluster.cc:1428] Setting key 63aa5ecf58dd3594f6ac8fdf52e2b9e4
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:24.182734 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:24.156775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:54697 (local address 127.19.246.254:36837)
0810 03:19:24.157322 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.157328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.158697 (+ 1369us) server_negotiation.cc:407] Connection header received
0810 03:19:24.160848 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.160854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.160936 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.161046 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.163024 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.164028 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.166025 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.166832 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.166965 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:24.172661 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:24.172667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:24.172671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:24.172762 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:24.177084 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.178849 (+ 1765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.178857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.178860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.179063 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.179908 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.179912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.179914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.180264 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:24.180439 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:24.181938 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0810 03:19:24.182350 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":256,"threads_started":1}
I20260810 03:19:24.184402 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:24.157497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36837 (local address 127.19.246.195:54697)
0810 03:19:24.158482 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:24.158543 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:24.160484 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:24.161380 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:24.161390 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:24.162096 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:24.162783 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:24.162806 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.164343 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.164347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:24.165803 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:24.165821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.167199 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.167207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:24.167577 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:24.168650 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:24.168708 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:24.172337 (+ 3629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:24.177475 (+ 5138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:24.177489 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:24.177514 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:24.178611 (+ 1097us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:24.179234 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:24.179238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:24.179241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:24.179753 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:24.180553 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:24.180564 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:24.181165 (+ 601us) client_negotiation.cc:769] Sending connection context
0810 03:19:24.181633 (+ 468us) client_negotiation.cc:240] Negotiation successful
0810 03:19:24.182276 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":437,"threads_started":1}
I20260810 03:19:24.186308 27092 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36837
I20260810 03:19:24.186715 27092 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:24.187881 27092 heartbeater.cc:511] Master 127.19.246.254:36837 requested a full tablet report, sending...
I20260810 03:19:24.189904 26586 ts_manager.cc:194] Registered new tserver with Master: a1ddf38af28749948c5b32e444bc8964 (127.19.246.195:33571)
I20260810 03:19:24.190912 26586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:54697
I20260810 03:19:24.201531 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/test-admin.keytab.
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:24.245048 20443 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 03:19:24.253749 27100 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:24.249418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.19.246.193:44789)
0810 03:19:24.250638 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.250646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.250693 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:19:24.250788 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.250795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.251142 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.251503 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.253254 (+ 1751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59266: BlockingRecv error: recv got EOF from 127.0.0.1:59266 (error 108)
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":604,"threads_started":1}
W20260810 03:19:24.254196 20443 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:24.256810 20443 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:24.258404 20443 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:19:24.284104 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:24.260553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.19.246.193:44789)
0810 03:19:24.260839 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.260846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.261180 (+ 334us) server_negotiation.cc:407] Connection header received
0810 03:19:24.261399 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.261404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.261476 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.261575 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.263028 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.264399 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.266556 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.267582 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.267760 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:24.273889 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:24.273919 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:24.273955 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:24.274033 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:24.279004 (+ 4971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.280468 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.280480 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.280483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.280739 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.281486 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.281492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.281495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.282034 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:24.282353 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:24.283188 (+ 835us) server_negotiation.cc:299] Negotiation successful
0810 03:19:24.283790 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260810 03:19:24.293277 20443 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5cc' of type FILE
W20260810 03:19:24.293270 27100 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:24.290419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.19.246.193:44789)
0810 03:19:24.290787 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.290792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.290961 (+ 169us) server_negotiation.cc:407] Connection header received
0810 03:19:24.291268 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.291274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.291567 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.291710 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.292804 (+ 1094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59286: BlockingRecv error: recv got EOF from 127.0.0.1:59286 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:24.295223 20443 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:24.296900 20443 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:19:24.322443 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:24.299279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.19.246.193:44789)
0810 03:19:24.299711 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.299720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.300337 (+ 617us) server_negotiation.cc:407] Connection header received
0810 03:19:24.300449 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.300453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.300532 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.300642 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.302276 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.303080 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.304991 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.305597 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.305759 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:24.312465 (+ 6706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:24.312472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:24.312477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:24.312550 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:24.318028 (+ 5478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.319102 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.319107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.319109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.319284 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.320156 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.320160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.320162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.320487 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:24.320810 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:24.321406 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 03:19:24.321943 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
W20260810 03:19:24.333240 27100 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:24.329550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.19.246.193:44789)
0810 03:19:24.330016 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.330025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.330369 (+ 344us) server_negotiation.cc:407] Connection header received
0810 03:19:24.330902 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.330911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.331025 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.331175 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.332872 (+ 1697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59304: BlockingRecv error: recv got EOF from 127.0.0.1:59304 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
W20260810 03:19:24.333604 20443 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786331905428759-20443-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:24.335887 20443 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:24.337788 20443 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:19:24.362429 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:24.340065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.19.246.193:44789)
0810 03:19:24.340436 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.340444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.340763 (+ 319us) server_negotiation.cc:407] Connection header received
0810 03:19:24.341021 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.341026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.341107 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.341237 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.342475 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.343414 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.346508 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.347465 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.347716 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:24.354620 (+ 6904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:24.354627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:24.354631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:24.354683 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:24.359249 (+ 4566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.360205 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.360211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.360214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.360412 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.360846 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.360860 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.360862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.361179 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:24.361522 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:24.361816 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 03:19:24.362077 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260810 03:19:24.364838 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 26654
I20260810 03:19:24.381481 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:24.365751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.19.246.193:44789)
0810 03:19:24.366031 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.366037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.366329 (+ 292us) server_negotiation.cc:407] Connection header received
0810 03:19:24.366455 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.366460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.366559 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.366696 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.368018 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.368951 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.370589 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.371114 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.371274 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:24.373778 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:24.373784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:24.373788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:24.373828 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:24.378205 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.379212 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.379217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.379219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.379434 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.379859 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.379862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.379864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.380160 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:24.380427 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:24.380798 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 03:19:24.380993 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260810 03:19:24.382262 26794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:24.476491 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 26654
I20260810 03:19:24.487886 26946 heartbeater.cc:503] Master 127.19.246.254:36837 was elected leader, sending a full tablet report...
I20260810 03:19:24.499576 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 26806
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:19:24.522455 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:24.501112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.19.246.194:40179)
0810 03:19:24.501782 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.501790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.501820 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:19:24.501895 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.501902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.502148 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.502326 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.504379 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.505551 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.507100 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.507861 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.507989 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:24.513956 (+ 5967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:24.513976 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:24.513993 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:24.514044 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:24.517736 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.519495 (+ 1759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.519522 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.519525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.519731 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.520629 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.520633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.520635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.521067 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:24.521338 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:24.521815 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 03:19:24.522064 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":288,"threads_started":1}
I20260810 03:19:24.524204 26941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:24.614070 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 26806
I20260810 03:19:24.634073 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 26952
Aug 10 03:19:24 dist-test-slave-btw8 krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:19:24.656565 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:24.635439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.19.246.195:33571)
0810 03:19:24.636349 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:24.636357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:24.636390 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:19:24.636467 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:24.636474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:24.636717 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:24.636871 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:24.638801 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.640268 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.642080 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:24.642880 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:24.643031 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:24.648860 (+ 5829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:24.648878 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:24.648896 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:24.648941 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:24.652587 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.653545 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.653553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.653555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.653848 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:24.654377 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:24.654381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:24.654383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:24.654961 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:24.655313 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:24.655845 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 03:19:24.656205 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":423,"threads_started":1}
I20260810 03:19:24.658264 27087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:24.755913 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 26952
I20260810 03:19:24.776782 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 26553
I20260810 03:19:24.778712 26620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:24.898733 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 26553
2026-08-10T03:19:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (5335 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:19:25 dist-test-slave-btw8 krb5kdc[27122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:19:25 dist-test-slave-btw8 krb5kdc[27122](info): set up 2 sockets
Aug 10 03:19:25 dist-test-slave-btw8 krb5kdc[27122](info): commencing operation
krb5kdc: starting...
W20260810 03:19:26.198582 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.005s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:19:26 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:19:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.254.keytab.
Entry for principal oryx/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.254.keytab.
I20260810 03:19:26.530934 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:34045
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.254.keytab
--principal=oryx/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:26.947582 27136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:26.948077 27136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:26.948287 27136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:26.963728 27136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:26.963920 27136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:26.964016 27136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:26.964121 27136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:26.964196 27136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:26.978732 27136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:34045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.254.keytab
--principal=oryx/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:34045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:26.980602 27136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:26.984308 27136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:26.999888 27141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:27.000079 27142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:27.000079 27144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:27.002097 27136 server_base.cc:1034] running on GCE node
I20260810 03:19:27.003213 27136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:27.006678 27136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:27.010251 27136 hybrid_clock.cc:648] HybridClock initialized: now 1786331967010124 us; error 99 us; skew 500 ppm
I20260810 03:19:27.010757 27136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:27 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331967, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:27.015484 27136 init.cc:377] Logged in from keytab as oryx/127.19.246.254@KRBTEST.COM (short username oryx)
I20260810 03:19:27.017807 27136 webserver.cc:460] Webserver started at http://127.19.246.254:46759/ using document root <none> and password file <none>
I20260810 03:19:27.018755 27136 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:27.018934 27136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:27.019330 27136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:27.022962 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "10020f23c05d4fecaa7a383cb192d439"
format_stamp: "Formatted at 2026-08-10 03:19:27 on dist-test-slave-btw8"
server_key: "6261fbaa3f2f0a13bc6c5ac40c38c3e6"
server_key_iv: "9464b3995feb2aca9785a30cf7094ea3"
server_key_version: "encryptionkey@0"
I20260810 03:19:27.024065 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "10020f23c05d4fecaa7a383cb192d439"
format_stamp: "Formatted at 2026-08-10 03:19:27 on dist-test-slave-btw8"
server_key: "6261fbaa3f2f0a13bc6c5ac40c38c3e6"
server_key_iv: "9464b3995feb2aca9785a30cf7094ea3"
server_key_version: "encryptionkey@0"
I20260810 03:19:27.032959 27136 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260810 03:19:27.037957 27151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:27.040656 27136 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260810 03:19:27.040894 27136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "10020f23c05d4fecaa7a383cb192d439"
format_stamp: "Formatted at 2026-08-10 03:19:27 on dist-test-slave-btw8"
server_key: "6261fbaa3f2f0a13bc6c5ac40c38c3e6"
server_key_iv: "9464b3995feb2aca9785a30cf7094ea3"
server_key_version: "encryptionkey@0"
I20260810 03:19:27.041203 27136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:27.069613 27136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:27.076756 27136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:27.077138 27136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:27.104102 27136 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:34045
I20260810 03:19:27.104147 27207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:34045 every 8 connection(s)
I20260810 03:19:27.105695 27136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:27.108840 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 27136
I20260810 03:19:27.109093 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:27.109961 20443 external_mini_cluster.cc:1428] Setting key 484bd18015052039964670ee2612e9cc
I20260810 03:19:27.110749 27208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:27.129730 27208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: Bootstrap starting.
Aug 10 03:19:27 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:27.133879 27208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:27.135339 27208 log.cc:824] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:27.139792 27208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: No bootstrap required, opened a new log
I20260810 03:19:27.144155 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:27.112422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.19.246.254:34045)
0810 03:19:27.113336 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:27.113350 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:27.113408 (+ 58us) server_negotiation.cc:407] Connection header received
0810 03:19:27.116293 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:27.116330 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:27.116922 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:27.117377 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:27.120101 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:27.121753 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:27.123974 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:27.124965 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:27.125170 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:27.131330 (+ 6160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:27.131340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:27.131437 (+ 97us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:27.131498 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:27.135825 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:27.137045 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:27.137054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:27.137063 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:27.137323 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:27.138046 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:27.138062 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:27.138064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:27.138877 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:27.140430 (+ 1553us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:27.141101 (+ 671us) server_negotiation.cc:299] Negotiation successful
0810 03:19:27.141693 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":416,"threads_started":1}
I20260810 03:19:27.155002 27208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } }
I20260810 03:19:27.155593 27208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:27.155719 27208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10020f23c05d4fecaa7a383cb192d439, State: Initialized, Role: FOLLOWER
I20260810 03:19:27.156539 27208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } }
I20260810 03:19:27.156870 27208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:27.157044 27208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:27.157256 27208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:27.159847 27208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } }
I20260810 03:19:27.160485 27208 leader_election.cc:302] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10020f23c05d4fecaa7a383cb192d439; no voters:
I20260810 03:19:27.161283 27208 leader_election.cc:288] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:27.161931 27215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:27.163410 27215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 1 LEADER]: Becoming Leader. State: Replica: 10020f23c05d4fecaa7a383cb192d439, State: Running, Role: LEADER
I20260810 03:19:27.164964 27215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } }
I20260810 03:19:27.165689 27208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:27.169241 27216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } }
I20260810 03:19:27.169198 27217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10020f23c05d4fecaa7a383cb192d439. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } }
I20260810 03:19:27.169920 27216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:27.169920 27217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:27.171141 27222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:27.177767 27222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:27.192479 27222 catalog_manager.cc:1347] Generated new cluster ID: 62ad2ac64ac9400cb63c1b9e00918d0b
I20260810 03:19:27.193240 27222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:27.207679 27222 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:27.209488 27222 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:27.230247 27222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: Generated new TSK 0
I20260810 03:19:27.231626 27222 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab.
Entry for principal oryx/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab.
I20260810 03:19:27.334112 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34045
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab
--principal=oryx/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:27.748200 27236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:27.748528 27236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:27.748693 27236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:27.756814 27236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:27.756959 27236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:27.757143 27236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:27.767917 27236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab
--principal=oryx/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:27.769598 27236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:27.771313 27236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:27.787089 27242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:27.787132 27241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:27.788030 27244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:27.788262 27236 server_base.cc:1034] running on GCE node
I20260810 03:19:27.788995 27236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:27.790145 27236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:27.791522 27236 hybrid_clock.cc:648] HybridClock initialized: now 1786331967791456 us; error 46 us; skew 500 ppm
I20260810 03:19:27.792097 27236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:27 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331967, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:27.796437 27236 init.cc:377] Logged in from keytab as oryx/127.19.246.193@KRBTEST.COM (short username oryx)
I20260810 03:19:27.798458 27236 webserver.cc:460] Webserver started at http://127.19.246.193:43945/ using document root <none> and password file <none>
I20260810 03:19:27.799607 27236 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:27.799796 27236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:27.800169 27236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:27.803678 27236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "b316fe967e474cd9806b4cb7e30af850"
format_stamp: "Formatted at 2026-08-10 03:19:27 on dist-test-slave-btw8"
server_key: "2ebec4478e5d626d6089652dc201e05d"
server_key_iv: "74d87874a5aad775a9558cd56a69a583"
server_key_version: "encryptionkey@0"
I20260810 03:19:27.804607 27236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "b316fe967e474cd9806b4cb7e30af850"
format_stamp: "Formatted at 2026-08-10 03:19:27 on dist-test-slave-btw8"
server_key: "2ebec4478e5d626d6089652dc201e05d"
server_key_iv: "74d87874a5aad775a9558cd56a69a583"
server_key_version: "encryptionkey@0"
I20260810 03:19:27.811203 27236 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:19:27.816860 27251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:27.818917 27236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260810 03:19:27.819278 27236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "b316fe967e474cd9806b4cb7e30af850"
format_stamp: "Formatted at 2026-08-10 03:19:27 on dist-test-slave-btw8"
server_key: "2ebec4478e5d626d6089652dc201e05d"
server_key_iv: "74d87874a5aad775a9558cd56a69a583"
server_key_version: "encryptionkey@0"
I20260810 03:19:27.819602 27236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:27.839092 27236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:27.844926 27236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:27.845306 27236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:27.846455 27236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:27.848170 27236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:27.848253 27236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:27.848348 27236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:27.848394 27236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:27.889348 27236 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:44087
I20260810 03:19:27.889386 27375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:44087 every 8 connection(s)
I20260810 03:19:27.890808 27236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:27.901072 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 27236
I20260810 03:19:27.901369 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:19:27.902441 20443 external_mini_cluster.cc:1428] Setting key 0494ee6da47748474aa34f07e82bca77
Aug 10 03:19:27 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331967, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:27.919061 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:27.894739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:56753 (local address 127.19.246.254:34045)
0810 03:19:27.895324 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:27.895329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:27.896664 (+ 1335us) server_negotiation.cc:407] Connection header received
0810 03:19:27.898815 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:27.898821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:27.898927 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:27.899062 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:27.901297 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:27.902672 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:27.905726 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:27.906364 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:27.906532 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:27.912320 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:27.912326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:27.912330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:27.912371 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:27.915283 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:27.916350 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:27.916359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:27.916365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:27.916541 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:27.917100 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:27.917113 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:27.917116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:27.917444 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:27.917618 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:27.918412 (+ 794us) server_negotiation.cc:299] Negotiation successful
0810 03:19:27.918667 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":321,"threads_started":1}
I20260810 03:19:27.920177 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:27.895575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.193:56753)
0810 03:19:27.896452 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:27.896498 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:27.898446 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:27.899416 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:27.899428 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:27.900106 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:27.900949 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:27.900983 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:27.902958 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:27.902967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:27.905495 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:27.905516 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:27.906691 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:27.906705 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:27.907103 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:27.908134 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:27.908182 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:27.911904 (+ 3722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:27.915557 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:27.915571 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:27.915597 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:27.916177 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:27.916728 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:27.916732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:27.916734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:27.916943 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:27.917588 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:27.917597 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:27.918019 (+ 422us) client_negotiation.cc:769] Sending connection context
0810 03:19:27.918310 (+ 291us) client_negotiation.cc:240] Negotiation successful
0810 03:19:27.918930 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":375,"threads_started":1}
I20260810 03:19:27.922102 27376 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34045
I20260810 03:19:27.922529 27376 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:27.923614 27376 heartbeater.cc:511] Master 127.19.246.254:34045 requested a full tablet report, sending...
I20260810 03:19:27.926728 27169 ts_manager.cc:194] Registered new tserver with Master: b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087)
I20260810 03:19:27.929212 27169 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:56753
WARNING: no policy specified for oryx/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab.
Entry for principal oryx/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab.
I20260810 03:19:27.972064 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34045
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab
--principal=oryx/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:28.403869 27383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:28.404250 27383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:28.404376 27383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:28.413844 27383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:28.414359 27383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:28.414589 27383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:28.426892 27383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab
--principal=oryx/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:28.428480 27383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:28.431092 27383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:28.449936 27388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:28.450089 27389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:28.450657 27391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:28.450912 27383 server_base.cc:1034] running on GCE node
I20260810 03:19:28.451740 27383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:28.452947 27383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:28.454299 27383 hybrid_clock.cc:648] HybridClock initialized: now 1786331968454178 us; error 83 us; skew 500 ppm
I20260810 03:19:28.454829 27383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:28 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331968, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:28.459213 27383 init.cc:377] Logged in from keytab as oryx/127.19.246.194@KRBTEST.COM (short username oryx)
I20260810 03:19:28.461700 27383 webserver.cc:460] Webserver started at http://127.19.246.194:40723/ using document root <none> and password file <none>
I20260810 03:19:28.462651 27383 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:28.462850 27383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:28.463260 27383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:28.467633 27383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "c3ee6f0413794d16a015e1d1befde0a6"
format_stamp: "Formatted at 2026-08-10 03:19:28 on dist-test-slave-btw8"
server_key: "7199e18df64b666a787130fd187cfeb6"
server_key_iv: "c97ec5bb5b0a3502c00f13aa40db11bf"
server_key_version: "encryptionkey@0"
I20260810 03:19:28.469389 27383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "c3ee6f0413794d16a015e1d1befde0a6"
format_stamp: "Formatted at 2026-08-10 03:19:28 on dist-test-slave-btw8"
server_key: "7199e18df64b666a787130fd187cfeb6"
server_key_iv: "c97ec5bb5b0a3502c00f13aa40db11bf"
server_key_version: "encryptionkey@0"
I20260810 03:19:28.477118 27383 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260810 03:19:28.482550 27398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:28.484699 27383 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 03:19:28.484973 27383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "c3ee6f0413794d16a015e1d1befde0a6"
format_stamp: "Formatted at 2026-08-10 03:19:28 on dist-test-slave-btw8"
server_key: "7199e18df64b666a787130fd187cfeb6"
server_key_iv: "c97ec5bb5b0a3502c00f13aa40db11bf"
server_key_version: "encryptionkey@0"
I20260810 03:19:28.485239 27383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:28.506927 27383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:28.513870 27383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:28.514292 27383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:28.515568 27383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:28.517515 27383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:28.517709 27383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:28.517903 27383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:28.517984 27383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:28.561653 27383 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:34505
I20260810 03:19:28.561679 27524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:34505 every 8 connection(s)
I20260810 03:19:28.563195 27383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:28.568240 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 27383
I20260810 03:19:28.568511 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:19:28.569211 20443 external_mini_cluster.cc:1428] Setting key 5bb3cba7dc614c40525b1ad73256d49c
Aug 10 03:19:28 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331968, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:28.594019 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:28.566928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:57325 (local address 127.19.246.254:34045)
0810 03:19:28.567618 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:28.567624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:28.569036 (+ 1412us) server_negotiation.cc:407] Connection header received
0810 03:19:28.571835 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:28.571842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:28.571948 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:28.572093 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:28.575079 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:28.576041 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:28.579038 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:28.579795 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:28.579926 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:28.585793 (+ 5867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:28.585799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:28.585803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:28.585848 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:28.589523 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:28.590936 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:28.590941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:28.590944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:28.591182 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:28.591850 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:28.591854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:28.591856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:28.592200 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:28.592332 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:28.593455 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0810 03:19:28.593704 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":424,"threads_started":1}
I20260810 03:19:28.595736 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:28.567817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.194:57325)
0810 03:19:28.568804 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:28.568858 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:28.571102 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:28.572712 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:28.572727 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:28.573675 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:28.574767 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:28.574801 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:28.576257 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:28.576266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:28.578751 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:28.578779 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:28.580123 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:28.580132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:28.580564 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:28.581677 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:28.581725 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:28.585438 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:28.589748 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:28.589762 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:28.589789 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:28.590681 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:28.591384 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:28.591392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:28.591398 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:28.591695 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:28.592374 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:28.592386 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:28.592930 (+ 544us) client_negotiation.cc:769] Sending connection context
0810 03:19:28.593291 (+ 361us) client_negotiation.cc:240] Negotiation successful
0810 03:19:28.594005 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":437,"threads_started":1}
I20260810 03:19:28.597810 27525 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34045
I20260810 03:19:28.598250 27525 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:28.599136 27525 heartbeater.cc:511] Master 127.19.246.254:34045 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.19.246.195@KRBTEST.COM; defaulting to no policy
I20260810 03:19:28.601649 27169 ts_manager.cc:194] Registered new tserver with Master: c3ee6f0413794d16a015e1d1befde0a6 (127.19.246.194:34505)
I20260810 03:19:28.602676 27169 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:57325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab.
Entry for principal oryx/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab.
I20260810 03:19:28.644172 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34045
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab
--principal=oryx/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:28.934144 27376 heartbeater.cc:503] Master 127.19.246.254:34045 was elected leader, sending a full tablet report...
W20260810 03:19:29.042532 27532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:29.042904 27532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:29.043082 27532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:29.050980 27532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:29.051120 27532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:29.051262 27532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:29.062094 27532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab
--principal=oryx/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:34045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:29.064196 27532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:29.065900 27532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:29.084849 27537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:29.084945 27538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:29.084899 27540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:29.088101 27532 server_base.cc:1034] running on GCE node
I20260810 03:19:29.089112 27532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:29.090484 27532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:29.091918 27532 hybrid_clock.cc:648] HybridClock initialized: now 1786331969091808 us; error 111 us; skew 500 ppm
I20260810 03:19:29.092381 27532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:29 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331969, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:29.097570 27532 init.cc:377] Logged in from keytab as oryx/127.19.246.195@KRBTEST.COM (short username oryx)
I20260810 03:19:29.100522 27532 webserver.cc:460] Webserver started at http://127.19.246.195:39391/ using document root <none> and password file <none>
I20260810 03:19:29.101485 27532 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:29.101683 27532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:29.102094 27532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:29.106011 27532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "a0827a2564a94e71b9bacc423b64d5d8"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "909b883c8dfd5f8362fc7bc6cbc5d8b7"
server_key_iv: "945a870e1170659f5bf00fa77842bdc1"
server_key_version: "encryptionkey@0"
I20260810 03:19:29.107095 27532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "a0827a2564a94e71b9bacc423b64d5d8"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "909b883c8dfd5f8362fc7bc6cbc5d8b7"
server_key_iv: "945a870e1170659f5bf00fa77842bdc1"
server_key_version: "encryptionkey@0"
I20260810 03:19:29.114550 27532 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260810 03:19:29.120702 27547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:29.122733 27532 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260810 03:19:29.123010 27532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "a0827a2564a94e71b9bacc423b64d5d8"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "909b883c8dfd5f8362fc7bc6cbc5d8b7"
server_key_iv: "945a870e1170659f5bf00fa77842bdc1"
server_key_version: "encryptionkey@0"
I20260810 03:19:29.123296 27532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:29.148075 27532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:29.154076 27532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:29.154569 27532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:29.155846 27532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:29.157470 27532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:29.157588 27532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:29.157780 27532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:29.157860 27532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:29.202296 27532 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:38391
I20260810 03:19:29.202365 27670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:38391 every 8 connection(s)
I20260810 03:19:29.204174 27532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:29.207612 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 27532
I20260810 03:19:29.207954 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:19:29.208506 20443 external_mini_cluster.cc:1428] Setting key bab1a216a7d775a948d651ece1eff29d
Aug 10 03:19:29 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331969, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:29.230854 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.207516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:39307 (local address 127.19.246.254:34045)
0810 03:19:29.208175 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:29.208183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:29.209379 (+ 1196us) server_negotiation.cc:407] Connection header received
0810 03:19:29.211412 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:29.211418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:29.211507 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:29.211638 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:29.213810 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.214605 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.216797 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.217257 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.217376 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.222470 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:29.222488 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:29.222492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:29.222544 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:29.226886 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.228173 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.228177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.228180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.228388 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.228965 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.228969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.228971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.229368 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:29.229493 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:29.230322 (+ 829us) server_negotiation.cc:299] Negotiation successful
0810 03:19:29.230531 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":330,"threads_started":1}
I20260810 03:19:29.232113 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.208195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.195:39307)
0810 03:19:29.209114 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:29.209176 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:29.211067 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:29.212047 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:29.212058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:29.212764 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:29.213532 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.213555 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.214867 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.214872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.216539 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.216564 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.217674 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.217680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.217941 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.218903 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:29.218934 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:29.222090 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:29.227132 (+ 5042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.227143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.227193 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.227940 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.228525 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.228528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.228530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.228817 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.229505 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:29.229511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:29.229917 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 03:19:29.230242 (+ 325us) client_negotiation.cc:240] Negotiation successful
0810 03:19:29.230721 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":369,"threads_started":1}
I20260810 03:19:29.234047 27671 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34045
I20260810 03:19:29.234745 27671 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:29.235771 27671 heartbeater.cc:511] Master 127.19.246.254:34045 requested a full tablet report, sending...
I20260810 03:19:29.237771 27169 ts_manager.cc:194] Registered new tserver with Master: a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391)
I20260810 03:19:29.238942 27169 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:39307
I20260810 03:19:29.239526 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.246.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.253.keytab.
Entry for principal oryx/127.19.246.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.253.keytab.
I20260810 03:19:29.313751 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.253:39327
--webserver_interface=127.19.246.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.253.keytab
--principal=oryx/127.19.246.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:34045,127.19.246.253:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:29.606524 27525 heartbeater.cc:503] Master 127.19.246.254:34045 was elected leader, sending a full tablet report...
W20260810 03:19:29.717097 27680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:29.717437 27680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:29.717535 27680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:29.727073 27680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:29.727311 27680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:29.727458 27680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:29.727594 27680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:29.727689 27680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:29.739987 27680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:34045,127.19.246.253:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.253.keytab
--principal=oryx/127.19.246.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.253:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.246.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
W20260810 03:19:29.740502 27680 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.246.254:34045,127.19.246.253:39327'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 03:19:29.741707 27680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:29.743199 27680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:29.759258 27686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:29.759523 27687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:29.759487 27689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:29.765100 27680 server_base.cc:1034] running on GCE node
I20260810 03:19:29.766247 27680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:29.768321 27680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:29.769949 27680 hybrid_clock.cc:648] HybridClock initialized: now 1786331969769857 us; error 158 us; skew 500 ppm
I20260810 03:19:29.770411 27680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:29 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331969, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:29.775278 27680 init.cc:377] Logged in from keytab as oryx/127.19.246.253@KRBTEST.COM (short username oryx)
I20260810 03:19:29.778250 27680 webserver.cc:460] Webserver started at http://127.19.246.253:35383/ using document root <none> and password file <none>
I20260810 03:19:29.779201 27680 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:29.779314 27680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:29.779795 27680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:29.783533 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data/instance:
uuid: "a70cafd63bc949f2a59ec75087135282"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "469ff4b68443f508ae947dd573f5b04b"
server_key_iv: "183415a98e746febe42144dde61b1792"
server_key_version: "encryptionkey@0"
I20260810 03:19:29.784535 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal/instance:
uuid: "a70cafd63bc949f2a59ec75087135282"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "469ff4b68443f508ae947dd573f5b04b"
server_key_iv: "183415a98e746febe42144dde61b1792"
server_key_version: "encryptionkey@0"
I20260810 03:19:29.791708 27680 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 03:19:29.797578 27696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:29.799986 27680 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.004s
I20260810 03:19:29.800225 27680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
uuid: "a70cafd63bc949f2a59ec75087135282"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "469ff4b68443f508ae947dd573f5b04b"
server_key_iv: "183415a98e746febe42144dde61b1792"
server_key_version: "encryptionkey@0"
I20260810 03:19:29.800478 27680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:29.833192 27680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:29.840018 27680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:29.840365 27680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:29.863560 27680 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.253:39327
I20260810 03:19:29.863605 27750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.253:39327 every 8 connection(s)
I20260810 03:19:29.865095 27680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
I20260810 03:19:29.869839 27752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:29.869870 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 27680
I20260810 03:19:29.881271 27752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:19:29 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.253@KRBTEST.COM
Aug 10 03:19:29 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331969, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.253@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:29.900876 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.872545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.19.246.253:39327)
0810 03:19:29.873531 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:29.873544 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:29.873593 (+ 49us) server_negotiation.cc:407] Connection header received
0810 03:19:29.875467 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:29.875496 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:29.876016 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:29.876432 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:29.878754 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.880472 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.882529 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.883614 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.883823 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.890906 (+ 7083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:29.890937 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:29.890961 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:29.891027 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:29.895314 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.896278 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.896287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.896308 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.896494 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.896932 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.896938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.896941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.897778 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:29.898156 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:29.898751 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 03:19:29.899233 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":466,"threads_started":1}
I20260810 03:19:29.908128 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.885282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.19.246.254:34045)
0810 03:19:29.885830 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:29.885838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:29.886583 (+ 745us) server_negotiation.cc:407] Connection header received
0810 03:19:29.886833 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:29.886840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:29.886940 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:29.887119 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:29.889962 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.891537 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.894119 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.894698 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.894863 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.899584 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:29.899604 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:29.899608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:29.899666 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:29.903722 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.905403 (+ 1681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.905410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.905414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.905613 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.906089 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.906096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.906099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.906682 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:29.906852 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:29.907547 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 03:19:29.907752 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260810 03:19:29.908607 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.885285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:40860)
0810 03:19:29.886384 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:29.886440 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:29.886674 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:29.887229 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:29.887236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:29.888539 (+ 1303us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:29.889503 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.889537 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.891914 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.891919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.893873 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.893894 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.894939 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.894948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.895176 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.895971 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:29.895990 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:29.899233 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:29.903934 (+ 4701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.903946 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.903954 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.905258 (+ 1304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.905756 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.905761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.905765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.905935 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.906869 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:29.906879 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:29.907147 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 03:19:29.907451 (+ 304us) client_negotiation.cc:240] Negotiation successful
0810 03:19:29.907927 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":549,"threads_started":1}
I20260810 03:19:29.911408 27752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:19:29 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331969, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.253@KRBTEST.COM for oryx/127.19.246.253@KRBTEST.COM
I20260810 03:19:29.927448 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.912210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.0.0.1:51722)
0810 03:19:29.912505 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:29.912525 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:29.912634 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:29.913104 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:29.913109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:29.913571 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:29.913751 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.913762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.914883 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.914888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.916129 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.916146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.916959 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.916964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.917103 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.917879 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:29.917898 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:29.920428 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:29.924471 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.924475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.924478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.925116 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.925712 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.925716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.925719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.925868 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.926495 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:29.926499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:29.926677 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 03:19:29.926807 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 03:19:29.927088 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260810 03:19:29.927799 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.912391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.19.246.253:39327)
0810 03:19:29.912575 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:29.912580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:29.912598 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:19:29.912850 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:29.912855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:29.912916 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:29.913001 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:29.913911 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.914694 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.916333 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.916768 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.916893 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.920703 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:29.920709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:29.920712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:29.920751 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:29.924154 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.925353 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.925359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.925361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.925504 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.925995 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.925999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.926002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.926353 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:29.926498 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:29.926940 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 03:19:29.927269 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":68}
I20260810 03:19:29.933542 27752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Bootstrap starting.
I20260810 03:19:29.937444 27752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:29.938802 27752 log.cc:824] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:29.942298 27752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: No bootstrap required, opened a new log
I20260810 03:19:29.951184 27752 raft_consensus.cc:348] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } }
I20260810 03:19:29.951844 27752 raft_consensus.cc:374] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:29.952028 27752 raft_consensus.cc:729] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70cafd63bc949f2a59ec75087135282, State: Initialized, Role: FOLLOWER
I20260810 03:19:29.952818 27752 consensus_queue.cc:260] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } }
I20260810 03:19:29.954100 27759 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } } }
I20260810 03:19:29.954437 27759 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:19:29.955312 27752 sys_catalog.cc:564] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:19:29.966264 27771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:19:29.966477 27771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:19:29.970690 27680 master_runner.cc:428] Detected that this master a70cafd63bc949f2a59ec75087135282 is joining an existing cluster
I20260810 03:19:29.970824 27680 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.246.253:39327
I20260810 03:19:29.993790 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.975904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.19.246.254:34045)
0810 03:19:29.976098 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:29.976103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:29.976968 (+ 865us) server_negotiation.cc:407] Connection header received
0810 03:19:29.977243 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:29.977249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:29.977389 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:29.977480 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:29.979420 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.980616 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.982569 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.983045 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.983200 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.985575 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:29.985583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:29.985588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:29.985636 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:29.989760 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.991058 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.991066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.991073 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.991284 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.991846 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.991852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.991856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.992289 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:29.992483 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:29.993054 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 03:19:29.993378 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:19:29.993860 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.975931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:40872)
0810 03:19:29.976818 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:29.976842 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:29.977055 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:29.977829 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:29.977834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:29.978696 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:29.979173 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.979193 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.980811 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.980816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.982392 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.982407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.983269 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.983276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.983496 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.984252 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:29.984272 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:29.985316 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:29.990033 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.990039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.990043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.990658 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.991472 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.991481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.991484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.991707 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.992465 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:29.992472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:29.992719 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 03:19:29.992904 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 03:19:29.993305 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":41,"thread_start_us":411,"threads_started":1}
I20260810 03:19:29.995841 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.977147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.0.0.1:51724)
0810 03:19:29.977967 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:29.977990 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:29.978232 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:29.978696 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:29.978704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:29.979575 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:29.979862 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.979871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.981329 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.981335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.983091 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:29.983107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.984625 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.984632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:29.984873 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.986548 (+ 1675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:29.986572 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:29.987824 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:29.992585 (+ 4761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.992589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.992591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.993080 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.993545 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:29.993550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:29.993554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:29.993715 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:29.994590 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:29.994595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:29.994768 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 03:19:29.994954 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 03:19:29.995292 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":65,"thread_start_us":431,"threads_started":1}
I20260810 03:19:29.996223 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:29.977196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.19.246.253:39327)
0810 03:19:29.977424 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:29.977431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:29.978198 (+ 767us) server_negotiation.cc:407] Connection header received
0810 03:19:29.978438 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:29.978444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:29.978536 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:29.978657 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:29.980005 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.981094 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.983268 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:29.984137 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:29.984280 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:29.988104 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:29.988114 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:29.988128 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:29.988178 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:29.992396 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.993217 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.993222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.993225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.993415 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:29.993851 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:29.993855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:29.993859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:29.994427 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:29.994642 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:29.995069 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 03:19:29.995870 (+ 801us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":72}
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331967, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.254@KRBTEST.COM for oryx/127.19.246.253@KRBTEST.COM
I20260810 03:19:30.027014 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.001622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.0.0.1:51726)
0810 03:19:30.002383 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.002421 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.002646 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.003762 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.003769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.004634 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.004940 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.004950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.006949 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.006954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.008609 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.008629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.010538 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.010545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.010864 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.012008 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.012041 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.016979 (+ 4938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.022758 (+ 5779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.022788 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.022798 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.023646 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.024314 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.024320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.024324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.024536 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.025560 (+ 1024us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.025571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.025840 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.026085 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.026532 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":362,"threads_started":1}
I20260810 03:19:30.027144 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.001726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.19.246.253:39327)
0810 03:19:30.001940 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.001948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.002542 (+ 594us) server_negotiation.cc:407] Connection header received
0810 03:19:30.002900 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.002907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.003035 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.003152 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.005130 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.006582 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.008911 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.009923 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.010109 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.017348 (+ 7239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.017358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.017367 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.017425 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.022449 (+ 5024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.023896 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.023903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.023906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.024129 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.024727 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.024736 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.024742 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.025340 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.025554 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.026302 (+ 748us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.026635 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:19:30.029951 27169 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.246.253:39327: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a70cafd63bc949f2a59ec75087135282"
member_type: NON_VOTER
last_known_addr {
host: "127.19.246.253"
port: 39327
}
attrs {
promote: true
}
}
dest_uuid: "10020f23c05d4fecaa7a383cb192d439"
cas_config_opid_index: -1
I20260810 03:19:30.031781 27169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.19.246.252@KRBTEST.COM; defaulting to no policy
I20260810 03:19:30.047328 27724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:30.046579 27782 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a70cafd63bc949f2a59ec75087135282 (127.19.246.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.252@KRBTEST.COM" created.
I20260810 03:19:30.055037 27724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Refusing update from remote peer 10020f23c05d4fecaa7a383cb192d439: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 03:19:30.054703 27782 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.246.253:39327
I20260810 03:19:30.055848 27782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } }
I20260810 03:19:30.056196 27782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:30.056257 27680 master.cc:537] Master@127.19.246.253:39327 shutting down...
I20260810 03:19:30.063669 27783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:19:30.074985 27153 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 03:19:30.075591 27680 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:30.076179 27680 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:30.076404 27680 tablet_replica.cc:330] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: stopping tablet replica
W20260810 03:19:30.077560 27153 consensus_peers.cc:487] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 -> Peer a70cafd63bc949f2a59ec75087135282 (127.19.246.253:39327): Couldn't send request to peer a70cafd63bc949f2a59ec75087135282. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:19:30.086220 27680 master.cc:552] Master@127.19.246.253:39327 shutdown complete.
I20260810 03:19:30.086416 27680 master_runner.cc:305] Clearing existing system tablet
I20260810 03:19:30.086983 27680 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:30.087108 27680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:30.091470 27680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:19:30.093705 27790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:30.094198 27680 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:30.094533 27680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:19:30.094642 27680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
uuid: "a70cafd63bc949f2a59ec75087135282"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "469ff4b68443f508ae947dd573f5b04b"
server_key_iv: "183415a98e746febe42144dde61b1792"
server_key_version: "encryptionkey@0"
I20260810 03:19:30.102355 27680 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.252.keytab.
Entry for principal oryx/127.19.246.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.252.keytab.
I20260810 03:19:30.109546 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.252:38643
--webserver_interface=127.19.246.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.252.keytab
--principal=oryx/127.19.246.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:34045,127.19.246.253:39327,127.19.246.252:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:30.112025 27680 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 03:19:30.112265 27680 log.cc:1196] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 03:19:30.112846 27680 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Deleting consensus metadata
I20260810 03:19:30.113170 27680 master_runner.cc:315] Copying system tablet from 127.19.246.254:34045
I20260810 03:19:30.131034 27680 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: tablet copy: Beginning tablet copy session from remote peer at address 127.19.246.254:34045
I20260810 03:19:30.152205 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.132808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.19.246.254:34045)
0810 03:19:30.133007 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.133014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.133177 (+ 163us) server_negotiation.cc:407] Connection header received
0810 03:19:30.133510 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.133516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.133618 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.133742 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.135285 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.136501 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.139601 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.140411 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.140614 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.143939 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.143948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.143953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.143999 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.148544 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.149611 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.149617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.149622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.149808 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.150317 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.150323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.150326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.150776 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.150935 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.151547 (+ 612us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.151815 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:19:30.152345 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.132315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:40882)
0810 03:19:30.133035 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.133067 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.133322 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.133823 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.133831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.134622 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.135013 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.135032 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.136728 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.136738 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.138994 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.139044 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.140711 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.140723 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.141050 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.141951 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.141997 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.143561 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.148806 (+ 5245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.148813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.148818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.149450 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.149961 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.149966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.149970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.150174 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.150935 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.150943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.151157 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.151378 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.151841 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":685,"threads_started":1}
I20260810 03:19:30.154223 27193 tablet_copy_service.cc:140] P 10020f23c05d4fecaa7a383cb192d439: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a70cafd63bc949f2a59ec75087135282 ({username='oryx', principal='oryx/127.19.246.253@KRBTEST.COM'} at 127.0.0.1:40882)
I20260810 03:19:30.154781 27193 tablet_copy_service.cc:161] P 10020f23c05d4fecaa7a383cb192d439: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a70cafd63bc949f2a59ec75087135282 at {username='oryx', principal='oryx/127.19.246.253@KRBTEST.COM'} at 127.0.0.1:40882: session id = a70cafd63bc949f2a59ec75087135282-00000000000000000000000000000000
I20260810 03:19:30.160864 27193 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 03:19:30.164814 27680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 03:19:30.173677 27680 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: tablet copy: Starting download of 0 data blocks...
I20260810 03:19:30.174206 27680 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: tablet copy: Starting download of 1 WAL segments...
I20260810 03:19:30.179320 27680 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 03:19:30.185550 27193 tablet_copy_service.cc:342] P 10020f23c05d4fecaa7a383cb192d439: Request end of tablet copy session a70cafd63bc949f2a59ec75087135282-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.246.253@KRBTEST.COM'} at 127.0.0.1:40882
I20260810 03:19:30.185923 27193 tablet_copy_service.cc:433] P 10020f23c05d4fecaa7a383cb192d439: ending tablet copy session a70cafd63bc949f2a59ec75087135282-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a70cafd63bc949f2a59ec75087135282
I20260810 03:19:30.200599 27680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:30.210155 27799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:30.211702 27800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:30.219796 27680 server_base.cc:1034] running on GCE node
W20260810 03:19:30.220062 27802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:30.220593 27680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:30.221382 27680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:30.222679 27680 hybrid_clock.cc:648] HybridClock initialized: now 1786331970222564 us; error 44 us; skew 500 ppm
I20260810 03:19:30.222946 27680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331970, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:30.225550 27680 init.cc:377] Logged in from keytab as oryx/127.19.246.253@KRBTEST.COM (short username oryx)
I20260810 03:19:30.233666 27680 webserver.cc:460] Webserver started at http://127.19.246.253:34569/ using document root <none> and password file <none>
I20260810 03:19:30.234375 27680 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:30.234508 27680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:30.239147 27680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:19:30.241567 27809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:30.242134 27680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 03:19:30.242379 27671 heartbeater.cc:503] Master 127.19.246.254:34045 was elected leader, sending a full tablet report...
I20260810 03:19:30.242306 27680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
uuid: "a70cafd63bc949f2a59ec75087135282"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "469ff4b68443f508ae947dd573f5b04b"
server_key_iv: "183415a98e746febe42144dde61b1792"
server_key_version: "encryptionkey@0"
I20260810 03:19:30.242515 27680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:30.269927 27680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:30.276788 27680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:30.307559 27680 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.253:39327
I20260810 03:19:30.307592 27863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.253:39327 every 8 connection(s)
I20260810 03:19:30.308882 27680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
I20260810 03:19:30.315595 27864 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:19:30.319872 27864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Bootstrap starting.
I20260810 03:19:30.342908 27864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:19:30.343889 27864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Bootstrap complete.
I20260810 03:19:30.345177 27864 raft_consensus.cc:348] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } }
I20260810 03:19:30.346055 27864 raft_consensus.cc:729] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a70cafd63bc949f2a59ec75087135282, State: Initialized, Role: LEARNER
I20260810 03:19:30.346506 27864 consensus_queue.cc:260] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } }
I20260810 03:19:30.347548 27867 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } }
I20260810 03:19:30.347962 27864 sys_catalog.cc:564] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:30.347967 27867 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:19:30.364187 27878 catalog_manager.cc:1259] Loaded cluster ID: 62ad2ac64ac9400cb63c1b9e00918d0b
I20260810 03:19:30.364336 27878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: loading cluster ID for follower catalog manager: success
I20260810 03:19:30.368979 27878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: acquiring CA information for follower catalog manager: success
I20260810 03:19:30.372454 27878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:19:30.497996 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.485106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.0.0.1:51732)
0810 03:19:30.485429 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.485451 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.485599 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.486814 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.486821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.486869 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:30.487088 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.487102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.490391 (+ 3289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.490399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.492758 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.492785 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.496529 (+ 3744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.496540 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.497017 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.497158 (+ 141us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.497372 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.497607 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260810 03:19:30.498680 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.485353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.19.246.253:39327)
0810 03:19:30.486302 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.486308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.486341 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:19:30.486431 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.486439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.486546 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.486696 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:30.487387 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.490125 (+ 2738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.493113 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.495845 (+ 2732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.496558 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.497011 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.497625 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.497865 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":518,"threads_started":1}
I20260810 03:19:30.500428 27837 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260810 03:19:30.501230 27867 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10020f23c05d4fecaa7a383cb192d439. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } }
I20260810 03:19:30.501582 27867 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: This master's current role is: LEARNER
W20260810 03:19:30.619659 27793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:30.620107 27793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:30.620277 27793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:30.629623 27793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:30.629908 27793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:30.630023 27793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:30.630149 27793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:30.630244 27793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:30.644131 27793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:34045,127.19.246.253:39327,127.19.246.252:38643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.252.keytab
--principal=oryx/127.19.246.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.252:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.246.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:30.645907 27793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:30.647541 27793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:30.662883 27885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:30.662956 27886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:30.662883 27888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:30.664469 27793 server_base.cc:1034] running on GCE node
I20260810 03:19:30.665773 27793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:30.668350 27793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:30.670051 27793 hybrid_clock.cc:648] HybridClock initialized: now 1786331970669983 us; error 63 us; skew 500 ppm
I20260810 03:19:30.670758 27793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331970, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:30.674930 27793 init.cc:377] Logged in from keytab as oryx/127.19.246.252@KRBTEST.COM (short username oryx)
I20260810 03:19:30.676966 27793 webserver.cc:460] Webserver started at http://127.19.246.252:41637/ using document root <none> and password file <none>
I20260810 03:19:30.677778 27793 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:30.677927 27793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:30.678282 27793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:30.682104 27793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data/instance:
uuid: "49b1c2860da94233aef20970525d2791"
format_stamp: "Formatted at 2026-08-10 03:19:30 on dist-test-slave-btw8"
server_key: "ba72d2d4eef9b23ae479e3772a965500"
server_key_iv: "82d5153f1ca210df079ff26c8aea76ef"
server_key_version: "encryptionkey@0"
I20260810 03:19:30.683034 27793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal/instance:
uuid: "49b1c2860da94233aef20970525d2791"
format_stamp: "Formatted at 2026-08-10 03:19:30 on dist-test-slave-btw8"
server_key: "ba72d2d4eef9b23ae479e3772a965500"
server_key_iv: "82d5153f1ca210df079ff26c8aea76ef"
server_key_version: "encryptionkey@0"
I20260810 03:19:30.689741 27793 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:19:30.695021 27896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:30.697137 27793 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260810 03:19:30.697546 27793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
uuid: "49b1c2860da94233aef20970525d2791"
format_stamp: "Formatted at 2026-08-10 03:19:30 on dist-test-slave-btw8"
server_key: "ba72d2d4eef9b23ae479e3772a965500"
server_key_iv: "82d5153f1ca210df079ff26c8aea76ef"
server_key_version: "encryptionkey@0"
I20260810 03:19:30.698048 27793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:30.716359 27793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:30.722262 27793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:30.722673 27793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:30.757835 27793 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.252:38643
I20260810 03:19:30.758057 27950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.252:38643 every 8 connection(s)
I20260810 03:19:30.759653 27793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
I20260810 03:19:30.765172 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 27793
I20260810 03:19:30.766199 27951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:30.780465 27951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.252@KRBTEST.COM
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331970, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.252@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:30.801189 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.768686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.19.246.252:38643)
0810 03:19:30.769973 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.769987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.770045 (+ 58us) server_negotiation.cc:407] Connection header received
0810 03:19:30.773095 (+ 3050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.773144 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.774033 (+ 889us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.774664 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.777817 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.780219 (+ 2402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.782867 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.784041 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.786194 (+ 2153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.791493 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.791504 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.791518 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.791569 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.795829 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.796883 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.796891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.796900 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.797084 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.797588 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.797592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.797594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.798423 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.798799 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.799427 (+ 628us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.799937 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":563,"threads_started":1}
I20260810 03:19:30.804518 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.784433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:40896)
0810 03:19:30.785421 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.785478 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.785729 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.786855 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.786866 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.788186 (+ 1320us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.788681 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.788699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.790104 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.790112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.791529 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.791549 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.792469 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.792478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.792686 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.793405 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.793424 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.796777 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.800844 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.800867 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.800875 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.801473 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.802059 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.802064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.802068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.802244 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.803254 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.803264 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.803571 (+ 307us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.803821 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.804171 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":647,"threads_started":1}
I20260810 03:19:30.804740 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.785316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.19.246.254:34045)
0810 03:19:30.786205 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.786209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.786226 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:19:30.786281 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.786284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.786344 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.786678 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.788969 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.789928 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.791753 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.792247 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.792398 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.797033 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.797038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.797042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.797082 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.800676 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.801693 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.801700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.801704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.801892 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.802400 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.802404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.802406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.802938 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.803179 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.804005 (+ 826us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.804300 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":664,"threads_started":1}
I20260810 03:19:30.807133 27951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331970, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.252@KRBTEST.COM for oryx/127.19.246.253@KRBTEST.COM
I20260810 03:19:30.826467 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.808303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.19.246.253:39327)
0810 03:19:30.808514 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.808521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.808544 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:19:30.808768 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.808773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.808847 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.808980 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.809946 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.810907 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.813365 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.813884 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.814058 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.819768 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.819774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.819778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.819821 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.822800 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.824000 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.824007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.824012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.824185 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.824691 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.824697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.824699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.825102 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.825298 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.825738 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.826013 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:19:30.826449 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.808046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.0.0.1:51746)
0810 03:19:30.808319 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.808363 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.808505 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.809008 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.809013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.809555 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.809729 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.809739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.811101 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.811110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.813124 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.813143 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.814143 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.814151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.814504 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.815949 (+ 1445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.815981 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.819474 (+ 3493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.823050 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.823057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.823061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.823842 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.824333 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.824341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.824347 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.824538 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.825250 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.825255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.825433 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.825595 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.826013 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 03:19:30.828346 27951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331970, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.252@KRBTEST.COM for oryx/127.19.246.252@KRBTEST.COM
I20260810 03:19:30.846581 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.829613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:38643 (local address 127.0.0.1:42732)
0810 03:19:30.829970 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.829992 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.830173 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.830495 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.830499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.830960 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.831112 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.831121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.832188 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.832197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.833075 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.833086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.833803 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.833809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.834040 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.834713 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.834732 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.837351 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.842601 (+ 5250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.842606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.842609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.843304 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.843946 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.843950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.843953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.844525 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.845371 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.845376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.845545 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.845803 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.846190 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260810 03:19:30.846903 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.829675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.19.246.252:38643)
0810 03:19:30.829971 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.829978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.830091 (+ 113us) server_negotiation.cc:407] Connection header received
0810 03:19:30.830298 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.830310 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.830365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.830471 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.831266 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.832057 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.833220 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.833630 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.833779 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.837662 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.837667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.837670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.837703 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.842381 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.843514 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.843524 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.843528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.843781 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.844835 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.844838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.844841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.845213 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.845347 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.845945 (+ 598us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.846456 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":206,"server-negotiator.queue_time_us":155}
I20260810 03:19:30.852948 27951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Bootstrap starting.
I20260810 03:19:30.857179 27951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:30.858485 27951 log.cc:824] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:30.862073 27951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: No bootstrap required, opened a new log
I20260810 03:19:30.870733 27951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } }
I20260810 03:19:30.871193 27951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:30.871807 27951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b1c2860da94233aef20970525d2791, State: Initialized, Role: FOLLOWER
I20260810 03:19:30.872843 27951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } }
I20260810 03:19:30.874408 27959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } } }
I20260810 03:19:30.875104 27959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:19:30.875753 27951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:19:30.888072 27973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:19:30.888293 27973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:19:30.893347 27793 master_runner.cc:428] Detected that this master 49b1c2860da94233aef20970525d2791 is joining an existing cluster
I20260810 03:19:30.893555 27793 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.246.252:38643
I20260810 03:19:30.910488 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 27236
I20260810 03:19:30.918627 27959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:30.919534 27959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } }
I20260810 03:19:30.921667 27959 leader_election.cc:288] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10020f23c05d4fecaa7a383cb192d439 (127.19.246.254:34045), a70cafd63bc949f2a59ec75087135282 (127.19.246.253:39327)
I20260810 03:19:30.922741 27180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49b1c2860da94233aef20970525d2791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10020f23c05d4fecaa7a383cb192d439" is_pre_election: true
I20260810 03:19:30.923636 27837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49b1c2860da94233aef20970525d2791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a70cafd63bc949f2a59ec75087135282" is_pre_election: true
I20260810 03:19:30.925743 27898 leader_election.cc:302] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49b1c2860da94233aef20970525d2791; no voters: 10020f23c05d4fecaa7a383cb192d439, a70cafd63bc949f2a59ec75087135282
I20260810 03:19:30.926750 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.899304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.19.246.254:34045)
0810 03:19:30.899769 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.899778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.901117 (+ 1339us) server_negotiation.cc:407] Connection header received
0810 03:19:30.901484 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.901491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.901608 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.901732 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.903782 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.905233 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.908369 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.909574 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.909771 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.917953 (+ 8182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.917964 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.917969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.918025 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.922537 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.923657 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.923665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.923670 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.923983 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.924531 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.924537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.924551 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.925016 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.925188 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.926075 (+ 887us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.926362 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260810 03:19:30.926772 27959 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 03:19:30.927971 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.899621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:40912)
0810 03:19:30.900961 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.900987 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.901279 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.902023 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.902028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.902929 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.903515 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.903533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.905544 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.905557 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.908057 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.908098 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.910357 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.910367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.910637 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.911694 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.911724 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.917606 (+ 5882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.922757 (+ 5151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.922764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.922771 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.923513 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.924140 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.924146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.924151 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.924381 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.925256 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.925261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.925630 (+ 369us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.925807 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.927298 (+ 1491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"mutex_wait_us":1027,"thread_start_us":925,"threads_started":1}
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.193@KRBTEST.COM
I20260810 03:19:30.941322 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.899518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.19.246.252:38643)
0810 03:19:30.899761 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.899770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.912144 (+ 12374us) server_negotiation.cc:407] Connection header received
0810 03:19:30.912477 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.912485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.912578 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.912723 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.923967 (+ 11244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.925159 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.927398 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.928308 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.928417 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.932417 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.932424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.932429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.932471 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.937422 (+ 4951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.938384 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.938390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.938393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.938578 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.939279 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.939288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.939293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.939861 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.940031 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.940636 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.940913 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 03:19:30.943115 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.900712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:38643 (local address 127.0.0.1:42738)
0810 03:19:30.912028 (+ 11316us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.912051 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.912267 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.912848 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.912852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.922451 (+ 9599us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.923728 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.923741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.925461 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.925469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.927162 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.927182 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.928835 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.928844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.929069 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.929893 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.929918 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.932196 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.937602 (+ 5406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.937608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.937614 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.938250 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.938744 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.938751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.938757 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.938927 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.940106 (+ 1179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.940111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.940323 (+ 212us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.942206 (+ 1883us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.942571 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11128,"thread_start_us":14226,"threads_started":1}
I20260810 03:19:30.948362 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.901803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.19.246.253:39327)
0810 03:19:30.902065 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.902074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.913789 (+ 11715us) server_negotiation.cc:407] Connection header received
0810 03:19:30.914108 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.914117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.914221 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.914307 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.919764 (+ 5457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.921036 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.929755 (+ 8719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.930454 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.930591 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.937395 (+ 6804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.937405 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.937412 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.937463 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.942096 (+ 4633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.943867 (+ 1771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.943875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.943880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.945733 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.946312 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.946318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.946322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.946741 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.946896 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.947640 (+ 744us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.947944 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:19:30.949944 27981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.901309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.0.0.1:51752)
0810 03:19:30.913654 (+ 12345us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.913679 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.913914 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.914627 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.914633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.917827 (+ 3194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.919482 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.919509 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.927420 (+ 7911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.927431 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.929514 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.929533 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.931073 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.931082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.931332 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.935525 (+ 4193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.935554 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.937076 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.942654 (+ 5578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.942670 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.942674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.943695 (+ 1021us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.945946 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.945952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.945956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.946153 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.946984 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.946990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.947243 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.948991 (+ 1748us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.949403 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12133,"mutex_wait_us":1,"thread_start_us":10261,"threads_started":1}
Aug 10 03:19:30 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331967, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.254@KRBTEST.COM for oryx/127.19.246.252@KRBTEST.COM
I20260810 03:19:30.962512 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.912719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.19.246.193:44087)
0810 03:19:30.913862 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.913874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.913911 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:19:30.913993 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.914004 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.914380 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.916636 (+ 2256us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.917042 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.929954 (+ 12912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.933001 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.934175 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.935025 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.951563 (+ 16538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.951605 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.951631 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.951689 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.956500 (+ 4811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.957850 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.957864 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.957871 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.958095 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.958653 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.958659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.958662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.959194 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.959565 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.961162 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.961981 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":2040,"threads_started":1}
I20260810 03:19:30.964363 27370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:30.973170 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.936204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:38643 (local address 127.0.0.1:42754)
0810 03:19:30.940164 (+ 3960us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:30.940188 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:30.940318 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:30.942828 (+ 2510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:30.942834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:30.943417 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:30.943667 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.943681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.946965 (+ 3284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.946974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.948953 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:30.949136 (+ 183us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.951098 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.951110 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:30.951443 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.952286 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:30.952316 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:30.957098 (+ 4782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:30.962385 (+ 5287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.962392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.962397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.963036 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.963637 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:30.963664 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:30.963668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:30.963844 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:30.971170 (+ 7326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:30.971177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:30.971726 (+ 549us) client_negotiation.cc:769] Sending connection context
0810 03:19:30.972087 (+ 361us) client_negotiation.cc:240] Negotiation successful
0810 03:19:30.972646 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3762}
I20260810 03:19:30.973241 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:30.940697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.19.246.252:38643)
0810 03:19:30.941665 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:30.941670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:30.941688 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:19:30.941795 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:30.941801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:30.941888 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:30.941996 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:30.945345 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.946731 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.949528 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:30.950786 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:30.956670 (+ 5884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:30.957401 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:30.957407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:30.957412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:30.957457 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:30.962153 (+ 4696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.963201 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.963210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.963214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.963494 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:30.970298 (+ 6804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:30.970309 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:30.970313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:30.970924 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:30.971186 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:30.972315 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0810 03:19:30.972696 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":16752,"threads_started":1}
W20260810 03:19:30.975634 27902 tablet.cc:2369] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:19:30.977412 27169 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.246.252:38643: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "49b1c2860da94233aef20970525d2791"
member_type: NON_VOTER
last_known_addr {
host: "127.19.246.252"
port: 38643
}
attrs {
promote: true
}
}
dest_uuid: "10020f23c05d4fecaa7a383cb192d439"
cas_config_opid_index: 5
I20260810 03:19:30.978823 27169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } }
I20260810 03:19:30.980827 27782 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 49b1c2860da94233aef20970525d2791 (127.19.246.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:30.995229 27925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:31.000213 27925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 FOLLOWER]: Refusing update from remote peer 10020f23c05d4fecaa7a383cb192d439: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 03:19:31.014307 27782 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.246.252:38643
I20260810 03:19:31.014919 27990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:31.015569 27793 master.cc:537] Master@127.19.246.252:38643 shutting down...
I20260810 03:19:31.018285 27837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 LEARNER]: Refusing update from remote peer 10020f23c05d4fecaa7a383cb192d439: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 03:19:31.021059 27782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 03:19:31.022083 27989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.029023 27989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
W20260810 03:19:31.030697 27152 consensus_peers.cc:487] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 -> Peer 49b1c2860da94233aef20970525d2791 (127.19.246.252:38643): Couldn't send request to peer 49b1c2860da94233aef20970525d2791. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:19:31.045063 27837 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 49b1c2860da94233aef20970525d2791 (127.19.246.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.052784 27997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: NON_VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: true } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.053328 27997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:19:31.054243 27989 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: attempting to promote NON_VOTER a70cafd63bc949f2a59ec75087135282 to VOTER
I20260810 03:19:31.056876 27989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } }
I20260810 03:19:31.060178 27793 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:31.060710 27793 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:31.060966 27793 tablet_replica.cc:330] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: stopping tablet replica
I20260810 03:19:31.065026 27837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 LEARNER]: Refusing update from remote peer 10020f23c05d4fecaa7a383cb192d439: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260810 03:19:31.065812 27152 consensus_peers.cc:487] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 -> Peer 49b1c2860da94233aef20970525d2791 (127.19.246.252:38643): Couldn't send request to peer 49b1c2860da94233aef20970525d2791. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:19:31.065997 27989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260810 03:19:31.070029 27782 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a70cafd63bc949f2a59ec75087135282 (127.19.246.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.075433 27793 master.cc:552] Master@127.19.246.252:38643 shutdown complete.
I20260810 03:19:31.075548 27793 master_runner.cc:305] Clearing existing system tablet
I20260810 03:19:31.075205 27992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.075657 27992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:31.076021 27793 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:31.076165 27793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:31.076411 27992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.076638 27992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:31.078073 27836 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a70cafd63bc949f2a59ec75087135282 (127.19.246.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.084142 27997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.084669 27997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:19:31.085707 27793 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.004s
I20260810 03:19:31.088049 27999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:31.088555 27793 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:31.088850 27793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 03:19:31.089037 27793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
uuid: "49b1c2860da94233aef20970525d2791"
format_stamp: "Formatted at 2026-08-10 03:19:30 on dist-test-slave-btw8"
server_key: "ba72d2d4eef9b23ae479e3772a965500"
server_key_iv: "82d5153f1ca210df079ff26c8aea76ef"
server_key_version: "encryptionkey@0"
I20260810 03:19:31.095598 27793 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:31.104887 27793 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 03:19:31.105158 27793 log.cc:1196] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 03:19:31.105916 27793 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Deleting consensus metadata
I20260810 03:19:31.106282 27793 master_runner.cc:315] Copying system tablet from 127.19.246.254:34045
I20260810 03:19:31.120786 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 27236
I20260810 03:19:31.150625 27793 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: tablet copy: Beginning tablet copy session from remote peer at address 127.19.246.254:34045
I20260810 03:19:31.159560 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:44087
--local_ip_for_outbound_sockets=127.19.246.193
--tserver_master_addrs=127.19.246.254:34045
--webserver_port=43945
--webserver_interface=127.19.246.193
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab
--principal=oryx/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327,127.19.246.252:38643 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:31.175448 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.151871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:40918)
0810 03:19:31.156583 (+ 4712us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:31.156617 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:31.156819 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:31.157315 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:31.157321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:31.157973 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:31.158322 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:31.158337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.159946 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.159952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:31.162336 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:31.162360 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.163604 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.163612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:31.163919 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:31.164792 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:31.164816 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:31.166090 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:31.171920 (+ 5830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:31.171927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:31.171932 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:31.172528 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:31.173051 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:31.173057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:31.173061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:31.173247 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:31.174132 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:31.174138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:31.174361 (+ 223us) client_negotiation.cc:769] Sending connection context
0810 03:19:31.174524 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 03:19:31.174915 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4494,"thread_start_us":4269,"threads_started":1}
I20260810 03:19:31.176004 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.151979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.19.246.254:34045)
0810 03:19:31.152335 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:31.152342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:31.156780 (+ 4438us) server_negotiation.cc:407] Connection header received
0810 03:19:31.156997 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:31.157006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:31.157116 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:31.157258 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:31.158541 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.159744 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.162598 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.163400 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.163590 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:31.166430 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:31.166442 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:31.166450 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:31.166536 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:31.171708 (+ 5172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:31.172705 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:31.172714 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:31.172720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:31.172898 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:31.173422 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:31.173432 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:31.173438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:31.173952 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:31.174151 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:31.174730 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 03:19:31.175068 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260810 03:19:31.176635 27193 tablet_copy_service.cc:140] P 10020f23c05d4fecaa7a383cb192d439: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 49b1c2860da94233aef20970525d2791 ({username='oryx', principal='oryx/127.19.246.252@KRBTEST.COM'} at 127.0.0.1:40918)
I20260810 03:19:31.176842 27193 tablet_copy_service.cc:161] P 10020f23c05d4fecaa7a383cb192d439: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 49b1c2860da94233aef20970525d2791 at {username='oryx', principal='oryx/127.19.246.252@KRBTEST.COM'} at 127.0.0.1:40918: session id = 49b1c2860da94233aef20970525d2791-00000000000000000000000000000000
I20260810 03:19:31.180796 27193 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 03:19:31.183254 27793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 03:19:31.193333 27793 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: tablet copy: Starting download of 0 data blocks...
I20260810 03:19:31.193873 27793 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: tablet copy: Starting download of 1 WAL segments...
I20260810 03:19:31.197139 27793 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 03:19:31.206453 27193 tablet_copy_service.cc:342] P 10020f23c05d4fecaa7a383cb192d439: Request end of tablet copy session 49b1c2860da94233aef20970525d2791-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.246.252@KRBTEST.COM'} at 127.0.0.1:40918
I20260810 03:19:31.206660 27193 tablet_copy_service.cc:433] P 10020f23c05d4fecaa7a383cb192d439: ending tablet copy session 49b1c2860da94233aef20970525d2791-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 49b1c2860da94233aef20970525d2791
I20260810 03:19:31.220747 27793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:31.224080 28007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:31.224860 28008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:31.235776 27793 server_base.cc:1034] running on GCE node
W20260810 03:19:31.237396 28010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:31.238221 27793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:31.239001 27793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:31.240232 27793 hybrid_clock.cc:648] HybridClock initialized: now 1786331971240164 us; error 70 us; skew 500 ppm
I20260810 03:19:31.240520 27793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:31 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331971, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:31.243117 27793 init.cc:377] Logged in from keytab as oryx/127.19.246.252@KRBTEST.COM (short username oryx)
I20260810 03:19:31.252591 27793 webserver.cc:460] Webserver started at http://127.19.246.252:37263/ using document root <none> and password file <none>
I20260810 03:19:31.253011 27793 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:31.253144 27793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:31.256830 27793 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:19:31.260380 28018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:31.261006 27793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:19:31.261155 27793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
uuid: "49b1c2860da94233aef20970525d2791"
format_stamp: "Formatted at 2026-08-10 03:19:30 on dist-test-slave-btw8"
server_key: "ba72d2d4eef9b23ae479e3772a965500"
server_key_iv: "82d5153f1ca210df079ff26c8aea76ef"
server_key_version: "encryptionkey@0"
I20260810 03:19:31.261372 27793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:31.292398 27793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:31.297711 27793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:31.364960 27793 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.252:38643
I20260810 03:19:31.365057 28074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.252:38643 every 8 connection(s)
I20260810 03:19:31.366245 27793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
I20260810 03:19:31.375458 28075 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:19:31.380154 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Bootstrap starting.
I20260810 03:19:31.404659 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:19:31.405313 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: Bootstrap complete.
I20260810 03:19:31.406544 28075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } }
I20260810 03:19:31.406924 28075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49b1c2860da94233aef20970525d2791, State: Initialized, Role: LEARNER
I20260810 03:19:31.407275 28075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } }
I20260810 03:19:31.408300 28078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.408874 28078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:19:31.409454 28075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:31.431638 28092 catalog_manager.cc:1259] Loaded cluster ID: 62ad2ac64ac9400cb63c1b9e00918d0b
I20260810 03:19:31.432008 28092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: loading cluster ID for follower catalog manager: success
I20260810 03:19:31.438560 28092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: acquiring CA information for follower catalog manager: success
I20260810 03:19:31.442415 28092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:19:31.690126 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.680396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.19.246.252:38643)
0810 03:19:31.681251 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:31.681256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:31.681278 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:19:31.681343 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:31.681347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:31.681418 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:31.681535 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:31.682087 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.684456 (+ 2369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.687491 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.688598 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.688822 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:31.688982 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:31.689540 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 03:19:31.689793 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":562,"threads_started":1}
I20260810 03:19:31.690161 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.680122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:38643 (local address 127.0.0.1:42770)
0810 03:19:31.680892 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:31.680917 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:31.681148 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:31.681613 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:31.681619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:31.681666 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:31.681883 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:31.681897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.684831 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.684839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:31.687076 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:31.687109 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.688894 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.688904 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:31.689239 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:31.689318 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 03:19:31.689572 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 03:19:31.689792 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":403,"threads_started":1}
I20260810 03:19:31.691637 28045 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 03:19:31.692443 28078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10020f23c05d4fecaa7a383cb192d439. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:31.692833 28078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: This master's current role is: LEARNER
W20260810 03:19:31.725594 28006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:31.726150 28006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:31.726509 28006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:31.737797 28006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:31.738147 28006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:31.738471 28006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:31.754747 28006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab
--principal=oryx/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:44087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=43945
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327,127.19.246.252:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:31.756639 28006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:31.758378 28006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:31.776820 28100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:31.777695 28101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:31.777730 28103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:31.779201 28006 server_base.cc:1034] running on GCE node
I20260810 03:19:31.780308 28006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:31.781739 28006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:31.783001 28006 hybrid_clock.cc:648] HybridClock initialized: now 1786331971782955 us; error 41 us; skew 500 ppm
I20260810 03:19:31.783526 28006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:31 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331971, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:31.789180 28006 init.cc:377] Logged in from keytab as oryx/127.19.246.193@KRBTEST.COM (short username oryx)
I20260810 03:19:31.791891 28006 webserver.cc:460] Webserver started at http://127.19.246.193:43945/ using document root <none> and password file <none>
I20260810 03:19:31.792872 28006 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:31.793095 28006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:31.801599 28006 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 03:19:31.807777 28111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:31.810298 28006 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260810 03:19:31.810673 28006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "b316fe967e474cd9806b4cb7e30af850"
format_stamp: "Formatted at 2026-08-10 03:19:27 on dist-test-slave-btw8"
server_key: "2ebec4478e5d626d6089652dc201e05d"
server_key_iv: "74d87874a5aad775a9558cd56a69a583"
server_key_version: "encryptionkey@0"
I20260810 03:19:31.812374 28006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:31.830958 28006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:31.837714 28006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:31.838124 28006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:31.839593 28006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:31.841522 28006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:31.841670 28006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:31.841866 28006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:31.842056 28006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:31.956561 28006 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:44087
I20260810 03:19:31.957049 28236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:44087 every 8 connection(s)
I20260810 03:19:31.960253 28006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:31.961742 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 28006
I20260810 03:19:31.962321 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 27383
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331971, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.252@KRBTEST.COM
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.194@KRBTEST.COM
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331971, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:32.037926 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.977693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:57027 (local address 127.19.246.252:38643)
0810 03:19:31.978025 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:31.978041 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:31.979258 (+ 1217us) server_negotiation.cc:407] Connection header received
0810 03:19:31.984433 (+ 5175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:31.984444 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:31.984606 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:31.984708 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:31.991085 (+ 6377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.993232 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.000087 (+ 6855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.001015 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.001141 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.019582 (+ 18441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:32.019595 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:32.019605 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:32.019889 (+ 284us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:32.025744 (+ 5855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.027460 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.027470 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.027477 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.027739 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.031861 (+ 4122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.031874 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.031881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.032553 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:32.032800 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:32.036477 (+ 3677us) server_negotiation.cc:299] Negotiation successful
0810 03:19:32.037006 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 03:19:32.041829 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.963984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.19.246.194:34505)
0810 03:19:31.977883 (+ 13899us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:31.977898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:31.977945 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:19:31.978057 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:31.978069 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:31.978418 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:31.978734 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:31.995945 (+ 17211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.998587 (+ 2642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.002573 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.004481 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.004644 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.026510 (+ 21866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:32.026695 (+ 185us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:32.026758 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:32.026870 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:32.034743 (+ 7873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.036239 (+ 1496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.036270 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.036278 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.036527 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.037130 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.037139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.037146 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.037761 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:32.038286 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:32.040745 (+ 2459us) server_negotiation.cc:299] Negotiation successful
0810 03:19:32.041120 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13614,"thread_start_us":14016,"threads_started":1}
I20260810 03:19:32.044642 27520 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331971, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.253@KRBTEST.COM
I20260810 03:19:32.056864 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.976288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:38643 (local address 127.19.246.193:57027)
0810 03:19:31.979037 (+ 2749us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:31.979113 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:31.983839 (+ 4726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:31.985668 (+ 1829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:31.985686 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:31.987576 (+ 1890us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:31.990408 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:31.990480 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:31.993716 (+ 3236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:31.993727 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:31.998271 (+ 4544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:31.998313 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.001782 (+ 3469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.001797 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.002395 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.008504 (+ 6109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:32.008554 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:32.019012 (+ 10458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:32.026064 (+ 7052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.026081 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.026114 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.027201 (+ 1087us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.027941 (+ 740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.027949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.027956 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.028370 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.032988 (+ 4618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:32.033004 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:32.033921 (+ 917us) client_negotiation.cc:769] Sending connection context
0810 03:19:32.038824 (+ 4903us) client_negotiation.cc:240] Negotiation successful
0810 03:19:32.040050 (+ 1226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2304,"thread_start_us":1745,"threads_started":1}
I20260810 03:19:32.061528 28237 heartbeater.cc:347] Connected to a master server at 127.19.246.252:38643
I20260810 03:19:32.062053 28237 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:32.063793 28237 heartbeater.cc:511] Master 127.19.246.252:38643 requested a full tablet report, sending...
I20260810 03:19:32.065416 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.009389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.19.246.193:50535)
0810 03:19:32.019201 (+ 9812us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:32.019239 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:32.019511 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:32.020457 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:32.020464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:32.022369 (+ 1905us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:32.023099 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.023118 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.026171 (+ 3053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.026228 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.029934 (+ 3706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.029972 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.041764 (+ 11792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.041777 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.042123 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.043034 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:32.043060 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:32.047500 (+ 4440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:32.055963 (+ 8463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.055974 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.055980 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.056565 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.061992 (+ 5427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.062000 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.062004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.062563 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.064106 (+ 1543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:32.064116 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:32.064338 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 03:19:32.064504 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 03:19:32.064925 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9571,"thread_start_us":8904,"threads_started":1}
I20260810 03:19:32.067487 28035 ts_manager.cc:194] Registered new tserver with Master: b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087)
I20260810 03:19:32.070439 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.988296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.193:51263)
0810 03:19:31.999479 (+ 11183us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:31.999644 (+ 165us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:31.999874 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:32.000746 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:32.000751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:32.001693 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:32.002280 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.002295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.004898 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.004905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.007830 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.007864 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.011256 (+ 3392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.011274 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.011908 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.016865 (+ 4957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:32.016912 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:32.036039 (+ 19127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:32.050165 (+ 14126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.050175 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.050182 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.060602 (+ 10420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.063647 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.063653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.063657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.067727 (+ 4070us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.068761 (+ 1034us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:32.068766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:32.069242 (+ 476us) client_negotiation.cc:769] Sending connection context
0810 03:19:32.069596 (+ 354us) client_negotiation.cc:240] Negotiation successful
0810 03:19:32.070024 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10892,"thread_start_us":10540,"threads_started":1}
I20260810 03:19:32.071441 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.008898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:50535 (local address 127.19.246.253:39327)
0810 03:19:32.015554 (+ 6656us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:32.015564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:32.019680 (+ 4116us) server_negotiation.cc:407] Connection header received
0810 03:19:32.019768 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:32.019774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:32.019894 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:32.020031 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:32.023396 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.025880 (+ 2484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.039720 (+ 13840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.040939 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.041108 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.047975 (+ 6867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:32.047987 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:32.047995 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:32.048113 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:32.055648 (+ 7535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.056926 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.056932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.056935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.059640 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.062855 (+ 3215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.062862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.062866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.063443 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:32.066238 (+ 2795us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:32.066433 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 03:19:32.066754 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6466,"thread_start_us":6258,"threads_started":1}
I20260810 03:19:32.072005 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:31.987686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:51263 (local address 127.19.246.254:34045)
0810 03:19:31.988958 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:31.988965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:31.999791 (+ 10826us) server_negotiation.cc:407] Connection header received
0810 03:19:32.000033 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:32.000039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:32.000193 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:32.000314 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:32.002544 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.004576 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.008150 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.010061 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.010255 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.042726 (+ 32471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:32.042738 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:32.042748 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:32.042818 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:32.049794 (+ 6976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.060932 (+ 11138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.060939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.060942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.061458 (+ 516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.067968 (+ 6510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.067975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.067981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.068485 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:32.068683 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:32.070980 (+ 2297us) server_negotiation.cc:299] Negotiation successful
0810 03:19:32.071214 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":1021,"threads_started":1}
I20260810 03:19:32.073591 28238 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34045
I20260810 03:19:32.073694 28243 heartbeater.cc:347] Connected to a master server at 127.19.246.253:39327
I20260810 03:19:32.074044 28238 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:32.074108 28243 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:32.075392 28243 heartbeater.cc:511] Master 127.19.246.253:39327 requested a full tablet report, sending...
I20260810 03:19:32.075392 28238 heartbeater.cc:511] Master 127.19.246.254:34045 requested a full tablet report, sending...
I20260810 03:19:32.077852 27169 ts_manager.cc:194] Re-registered known tserver with Master: b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087)
I20260810 03:19:32.078526 27827 ts_manager.cc:194] Registered new tserver with Master: b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087)
I20260810 03:19:32.079288 27169 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:51263
I20260810 03:19:32.187423 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 27383
I20260810 03:19:32.211989 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:34505
--local_ip_for_outbound_sockets=127.19.246.194
--tserver_master_addrs=127.19.246.254:34045
--webserver_port=40723
--webserver_interface=127.19.246.194
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab
--principal=oryx/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327,127.19.246.252:38643 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:32.286324 27989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: NON_VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: true } } }
I20260810 03:19:32.286648 27989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:32.288348 28252 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: attempting to promote NON_VOTER 49b1c2860da94233aef20970525d2791 to VOTER
I20260810 03:19:32.289038 28252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } }
I20260810 03:19:32.291445 27836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Refusing update from remote peer 10020f23c05d4fecaa7a383cb192d439: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 03:19:32.291478 28045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 LEARNER]: Refusing update from remote peer 10020f23c05d4fecaa7a383cb192d439: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 03:19:32.292179 28251 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 03:19:32.292680 28252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 03:19:32.296432 28252 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49b1c2860da94233aef20970525d2791 (127.19.246.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:32.297328 27836 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49b1c2860da94233aef20970525d2791 (127.19.246.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:32.303315 28252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:32.303740 28251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:32.304207 28251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:32.304597 28252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:32.309196 28258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:32.309901 28258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:19:32.311045 28251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:32.311523 28251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:32.312063 28045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49b1c2860da94233aef20970525d2791 (127.19.246.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:32.319972 28259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10020f23c05d4fecaa7a383cb192d439" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:32.320387 28259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 03:19:32.630658 28250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:32.630997 28250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:32.631168 28250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:32.639245 28250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:32.639428 28250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:32.639616 28250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:32.651775 28250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab
--principal=oryx/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:34505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=40723
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327,127.19.246.252:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:32.653394 28250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:32.654870 28250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:32.671821 28267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:32.671775 28265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:32.671891 28264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:32.677728 28250 server_base.cc:1034] running on GCE node
I20260810 03:19:32.678583 28250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:32.679934 28250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:32.681252 28250 hybrid_clock.cc:648] HybridClock initialized: now 1786331972681145 us; error 94 us; skew 500 ppm
I20260810 03:19:32.681728 28250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331972, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:32.686112 28250 init.cc:377] Logged in from keytab as oryx/127.19.246.194@KRBTEST.COM (short username oryx)
I20260810 03:19:32.688244 28250 webserver.cc:460] Webserver started at http://127.19.246.194:40723/ using document root <none> and password file <none>
I20260810 03:19:32.689098 28250 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:32.689365 28250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:32.696676 28250 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:19:32.700627 28276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:32.702512 28250 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:19:32.702764 28250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "c3ee6f0413794d16a015e1d1befde0a6"
format_stamp: "Formatted at 2026-08-10 03:19:28 on dist-test-slave-btw8"
server_key: "7199e18df64b666a787130fd187cfeb6"
server_key_iv: "c97ec5bb5b0a3502c00f13aa40db11bf"
server_key_version: "encryptionkey@0"
I20260810 03:19:32.704140 28250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:32.726656 28250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:32.733094 28250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:32.733613 28250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:32.735146 28250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:32.738281 28250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:32.738435 28250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:32.738538 28250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:32.738590 28250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:32.785307 28250 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:34505
I20260810 03:19:32.785491 28401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:34505 every 8 connection(s)
I20260810 03:19:32.786903 28250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:32.796015 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 28250
I20260810 03:19:32.796427 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 27532
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331972, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.253@KRBTEST.COM
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331972, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331972, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.252@KRBTEST.COM
I20260810 03:19:32.823591 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.790627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:46433 (local address 127.19.246.253:39327)
0810 03:19:32.791401 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:32.791407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:32.792595 (+ 1188us) server_negotiation.cc:407] Connection header received
0810 03:19:32.795322 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:32.795328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:32.795466 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:32.795564 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:32.798689 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.800119 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.804128 (+ 4009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.805205 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.805330 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.813980 (+ 8650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:32.813988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:32.813992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:32.814055 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:32.817921 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.819622 (+ 1701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.819628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.819631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.819791 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.820585 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.820597 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.820601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.821021 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:32.821228 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:32.822725 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0810 03:19:32.823024 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":499,"threads_started":1}
Aug 10 03:19:32 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.195@KRBTEST.COM
I20260810 03:19:32.825484 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.791270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.19.246.194:46433)
0810 03:19:32.792391 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:32.792443 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:32.794908 (+ 2465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:32.796171 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:32.796185 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:32.797283 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:32.798446 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.798483 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.800381 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.800385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.803903 (+ 3518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.803922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.808101 (+ 4179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.808111 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.808310 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.809028 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:32.809052 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:32.813557 (+ 4505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:32.818125 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.818143 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.818169 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.819444 (+ 1275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.819943 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.819949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.819952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.820444 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.821240 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:32.821252 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:32.821984 (+ 732us) client_negotiation.cc:769] Sending connection context
0810 03:19:32.822587 (+ 603us) client_negotiation.cc:240] Negotiation successful
0810 03:19:32.823513 (+ 926us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":94,"spinlock_wait_cycles":43776,"thread_start_us":475,"threads_started":1}
I20260810 03:19:32.827669 28405 heartbeater.cc:347] Connected to a master server at 127.19.246.253:39327
I20260810 03:19:32.828133 28405 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:32.830184 28405 heartbeater.cc:511] Master 127.19.246.253:39327 requested a full tablet report, sending...
I20260810 03:19:32.832813 27827 ts_manager.cc:194] Registered new tserver with Master: c3ee6f0413794d16a015e1d1befde0a6 (127.19.246.194:34505)
I20260810 03:19:32.833866 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.790487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:41903 (local address 127.19.246.252:38643)
0810 03:19:32.791145 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:32.791152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:32.792607 (+ 1455us) server_negotiation.cc:407] Connection header received
0810 03:19:32.795322 (+ 2715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:32.795328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:32.795466 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:32.795559 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:32.798885 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.800160 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.804548 (+ 4388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.805446 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.805570 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.815758 (+ 10188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:32.815765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:32.815769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:32.815821 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:32.821394 (+ 5573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.822981 (+ 1587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.822991 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.822997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.823562 (+ 565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.824369 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.824376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.824381 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.825045 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:32.825202 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:32.832740 (+ 7538us) server_negotiation.cc:299] Negotiation successful
0810 03:19:32.833176 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":302,"threads_started":1}
I20260810 03:19:32.838174 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.793095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.194:44435)
0810 03:19:32.793817 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:32.793849 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:32.794928 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:32.797721 (+ 2793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:32.797726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:32.799858 (+ 2132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:32.801313 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.801336 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.802978 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.802985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.805248 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.805278 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.809297 (+ 4019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.809306 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.809554 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.810228 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:32.810251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:32.814117 (+ 3866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:32.830924 (+ 16807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.830932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.830938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.831935 (+ 997us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.834998 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.835007 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.835013 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.835434 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.836291 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:32.836296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:32.836498 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 03:19:32.836699 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 03:19:32.837069 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":1,"thread_start_us":323,"threads_started":1}
I20260810 03:19:32.838970 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.793259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:44435 (local address 127.19.246.254:34045)
0810 03:19:32.797329 (+ 4070us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:32.797335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:32.797357 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:19:32.797422 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:32.797430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:32.797519 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:32.797629 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:32.801544 (+ 3915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.802747 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.807100 (+ 4353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.808071 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.808239 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.815450 (+ 7211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:32.815458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:32.815462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:32.815533 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:32.830583 (+ 15050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.834350 (+ 3767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.834360 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.834366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.834817 (+ 451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.835569 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.835574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.835577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.836136 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:32.836318 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:32.837837 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0810 03:19:32.838514 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3927,"thread_start_us":4610,"threads_started":1}
I20260810 03:19:32.839542 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.791588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:38643 (local address 127.19.246.194:41903)
0810 03:19:32.792389 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:32.792447 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:32.794908 (+ 2461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:32.796217 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:32.796221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:32.798241 (+ 2020us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:32.798658 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.798676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.800387 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.800395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.804338 (+ 3943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:32.804352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.806026 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.806036 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:32.806448 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.808606 (+ 2158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:32.808645 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:32.815094 (+ 6449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:32.821691 (+ 6597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.821701 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.821708 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.822766 (+ 1058us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.823828 (+ 1062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:32.823835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:32.823838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:32.824126 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:32.825343 (+ 1217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:32.825351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:32.832184 (+ 6833us) client_negotiation.cc:769] Sending connection context
0810 03:19:32.834511 (+ 2327us) client_negotiation.cc:240] Negotiation successful
0810 03:19:32.835609 (+ 1098us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":1656,"thread_start_us":458,"threads_started":1}
I20260810 03:19:32.841339 28403 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34045
I20260810 03:19:32.841589 28403 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:32.842587 28403 heartbeater.cc:511] Master 127.19.246.254:34045 requested a full tablet report, sending...
I20260810 03:19:32.842903 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:32.798374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.19.246.195:38391)
0810 03:19:32.802180 (+ 3806us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:32.802188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:32.802216 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:19:32.802274 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:32.802280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:32.802505 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:32.802708 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:32.805660 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.806924 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.810762 (+ 3838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:32.814241 (+ 3479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:32.814396 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:32.827951 (+ 13555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:32.827978 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:32.827999 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:32.828057 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:32.835257 (+ 7200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.836774 (+ 1517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.836785 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.836789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.837059 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:32.838090 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:32.838099 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:32.838106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:32.839105 (+ 999us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:32.839653 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:32.841749 (+ 2096us) server_negotiation.cc:299] Negotiation successful
0810 03:19:32.842217 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3613,"thread_start_us":3274,"threads_started":1}
I20260810 03:19:32.844607 28402 heartbeater.cc:347] Connected to a master server at 127.19.246.252:38643
I20260810 03:19:32.844982 28402 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:32.846844 28402 heartbeater.cc:511] Master 127.19.246.252:38643 requested a full tablet report, sending...
I20260810 03:19:32.848999 28035 ts_manager.cc:194] Registered new tserver with Master: c3ee6f0413794d16a015e1d1befde0a6 (127.19.246.194:34505)
I20260810 03:19:32.849064 27666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:32.852439 27169 ts_manager.cc:194] Re-registered known tserver with Master: c3ee6f0413794d16a015e1d1befde0a6 (127.19.246.194:34505)
I20260810 03:19:32.854143 27169 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:44435
I20260810 03:19:32.950771 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 27532
I20260810 03:19:32.974643 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:38391
--local_ip_for_outbound_sockets=127.19.246.195
--tserver_master_addrs=127.19.246.254:34045
--webserver_port=39391
--webserver_interface=127.19.246.195
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab
--principal=oryx/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327,127.19.246.252:38643 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:33.084022 28238 heartbeater.cc:503] Master 127.19.246.254:34045 was elected leader, sending a full tablet report...
W20260810 03:19:33.524483 28419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:33.524837 28419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:33.525010 28419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:33.533300 28419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:33.533579 28419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:33.533860 28419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:33.545508 28419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab
--principal=oryx/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:38391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=39391
--tserver_master_addrs=127.19.246.254:34045,127.19.246.253:39327,127.19.246.252:38643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:33.547290 28419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:33.548993 28419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:33.567035 28428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:33.567236 28424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:33.567236 28426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:33.568257 28419 server_base.cc:1034] running on GCE node
I20260810 03:19:33.569101 28419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:33.570411 28419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:33.571867 28419 hybrid_clock.cc:648] HybridClock initialized: now 1786331973571798 us; error 95 us; skew 500 ppm
I20260810 03:19:33.572726 28419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:33 dist-test-slave-btw8 krb5kdc[27122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331973, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:33.577009 28419 init.cc:377] Logged in from keytab as oryx/127.19.246.195@KRBTEST.COM (short username oryx)
I20260810 03:19:33.578908 28419 webserver.cc:460] Webserver started at http://127.19.246.195:39391/ using document root <none> and password file <none>
I20260810 03:19:33.579903 28419 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:33.580092 28419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:33.587616 28419 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 03:19:33.591584 28436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:33.593559 28419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:33.593844 28419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "a0827a2564a94e71b9bacc423b64d5d8"
format_stamp: "Formatted at 2026-08-10 03:19:29 on dist-test-slave-btw8"
server_key: "909b883c8dfd5f8362fc7bc6cbc5d8b7"
server_key_iv: "945a870e1170659f5bf00fa77842bdc1"
server_key_version: "encryptionkey@0"
I20260810 03:19:33.595237 28419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:33.645522 28419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:33.651587 28419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:33.652070 28419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:33.653632 28419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:33.655478 28419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:33.655599 28419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:33.655689 28419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:33.655738 28419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:33.699728 28419 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:38391
I20260810 03:19:33.699848 28564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:38391 every 8 connection(s)
I20260810 03:19:33.701603 28419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:33.703583 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 28419
Aug 10 03:19:33 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331973, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
Aug 10 03:19:33 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331973, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.253@KRBTEST.COM
Aug 10 03:19:33 dist-test-slave-btw8 krb5kdc[27122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331973, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.252@KRBTEST.COM
I20260810 03:19:33.756985 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.705199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:59173 (local address 127.19.246.254:34045)
0810 03:19:33.724193 (+ 18994us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.724200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.724222 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:19:33.724296 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.724304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.724404 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.724559 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:33.727965 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.729663 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.733395 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.734148 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.734307 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.740529 (+ 6222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:33.740539 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:33.740544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:33.740598 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:33.748881 (+ 8283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.750798 (+ 1917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.750810 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.750816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.751109 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.752097 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.752107 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.752114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.752728 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:33.752975 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.755678 (+ 2703us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.756317 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18865,"thread_start_us":18753,"threads_started":1}
I20260810 03:19:33.757192 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.706740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.195:59173)
0810 03:19:33.715371 (+ 8631us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:33.715457 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:33.722994 (+ 7537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:33.724945 (+ 1951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:33.724964 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:33.726335 (+ 1371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:33.727660 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.727701 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.729900 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.729906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.733093 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.733113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.734711 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.734720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.735262 (+ 542us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.736297 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:33.736340 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:33.740168 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:33.749099 (+ 8931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:33.749105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:33.749109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:33.750282 (+ 1173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:33.751494 (+ 1212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:33.751503 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:33.751510 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:33.751900 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:33.753250 (+ 1350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:33.753265 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:33.753928 (+ 663us) client_negotiation.cc:769] Sending connection context
0810 03:19:33.754391 (+ 463us) client_negotiation.cc:240] Negotiation successful
0810 03:19:33.755240 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8274,"mutex_wait_us":112,"thread_start_us":8855,"threads_started":1}
I20260810 03:19:33.759259 28566 heartbeater.cc:347] Connected to a master server at 127.19.246.254:34045
I20260810 03:19:33.759690 28566 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:33.760219 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.705076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:44969 (local address 127.19.246.252:38643)
0810 03:19:33.723685 (+ 18609us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.723696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.723738 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:19:33.726559 (+ 2821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.726568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.726678 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.726838 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:33.730405 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.732163 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.735638 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.736488 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.736612 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.742705 (+ 6093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:33.742717 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:33.742724 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:33.742790 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:33.748620 (+ 5830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.750945 (+ 2325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.750955 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.750961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.751269 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.752456 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.752465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.752471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.752944 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:33.753138 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.759506 (+ 6368us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.759829 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18375,"thread_start_us":19620,"threads_started":1}
I20260810 03:19:33.760784 28566 heartbeater.cc:511] Master 127.19.246.254:34045 requested a full tablet report, sending...
I20260810 03:19:33.761729 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.705999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:38643 (local address 127.19.246.195:44969)
0810 03:19:33.716370 (+ 10371us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:33.716395 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:33.725761 (+ 9366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:33.727151 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:33.727161 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:33.729779 (+ 2618us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:33.730154 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.730168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.732438 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.732449 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.735336 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.735394 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.737028 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.737037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.737271 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.738010 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:33.738050 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:33.742347 (+ 4297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:33.748934 (+ 6587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:33.748948 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:33.748974 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:33.750753 (+ 1779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:33.752065 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:33.752073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:33.752080 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:33.752287 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:33.758008 (+ 5721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:33.758017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:33.758322 (+ 305us) client_negotiation.cc:769] Sending connection context
0810 03:19:33.758513 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 03:19:33.761119 (+ 2606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8469,"mutex_wait_us":42,"thread_start_us":8157,"threads_started":1}
I20260810 03:19:33.762925 27169 ts_manager.cc:194] Re-registered known tserver with Master: a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391)
I20260810 03:19:33.763007 28565 heartbeater.cc:347] Connected to a master server at 127.19.246.252:38643
I20260810 03:19:33.763182 28565 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:33.763772 28565 heartbeater.cc:511] Master 127.19.246.252:38643 requested a full tablet report, sending...
I20260810 03:19:33.764590 27169 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:59173
I20260810 03:19:33.765235 28035 ts_manager.cc:194] Registered new tserver with Master: a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391)
I20260810 03:19:33.769196 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.716571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:59985 (local address 127.19.246.253:39327)
0810 03:19:33.729254 (+ 12683us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.729264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.729291 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:19:33.729363 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.729370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.729461 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.729587 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:33.731992 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.733158 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.735581 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.736397 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.737362 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.749287 (+ 11925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:33.749299 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:33.749305 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:33.749418 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:33.761063 (+ 11645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.766541 (+ 5478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.766551 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.766558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.766851 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.767472 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.767478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.767481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.767902 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:33.768343 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.768489 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.768773 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12518,"thread_start_us":12361,"threads_started":1}
I20260810 03:19:33.769229 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.718445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.19.246.195:59985)
0810 03:19:33.728459 (+ 10014us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:33.728486 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:33.728657 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:33.730363 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:33.730368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:33.731088 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:33.731662 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.731681 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.733355 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.733361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.735324 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.735341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.736690 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.736699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.736974 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.737694 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:33.737722 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:33.741341 (+ 3619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:33.761334 (+ 19993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:33.761351 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:33.761358 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:33.766158 (+ 4800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:33.767022 (+ 864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:33.767028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:33.767032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:33.767276 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:33.768056 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:33.768061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:33.768235 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 03:19:33.768390 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 03:19:33.768773 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9828,"mutex_wait_us":178,"thread_start_us":9605,"threads_started":1}
I20260810 03:19:33.770263 28567 heartbeater.cc:347] Connected to a master server at 127.19.246.253:39327
I20260810 03:19:33.770460 28567 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:33.770888 28567 heartbeater.cc:511] Master 127.19.246.253:39327 requested a full tablet report, sending...
I20260810 03:19:33.772418 27827 ts_manager.cc:194] Registered new tserver with Master: a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391)
I20260810 03:19:33.794480 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.774071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.19.246.253:39327)
0810 03:19:33.774810 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.774815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.775916 (+ 1101us) server_negotiation.cc:407] Connection header received
0810 03:19:33.776130 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.776136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.776215 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.776324 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:33.779695 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.780650 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.782457 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.783321 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.783530 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.786732 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:33.786744 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:33.786749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:33.786799 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:33.790608 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.791802 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.791809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.791813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.791991 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.792460 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.792465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.792469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.792899 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:33.793203 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.793925 (+ 722us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.794172 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623}
I20260810 03:19:33.794979 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.748898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.19.246.252:38643)
0810 03:19:33.760585 (+ 11687us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.760592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.773883 (+ 13291us) server_negotiation.cc:407] Connection header received
0810 03:19:33.774117 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.774123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.774232 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.774928 (+ 696us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:33.778125 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.778888 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.780969 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.781997 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.782177 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.785926 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:33.785935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:33.785942 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:33.785994 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:33.790958 (+ 4964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.792025 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.792030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.792034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.792216 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.792683 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.792687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.792690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.793149 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:33.793414 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.793972 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.794487 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11572,"thread_start_us":22803,"threads_started":1}
I20260810 03:19:33.797864 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.741474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.19.246.254:34045)
0810 03:19:33.757561 (+ 16087us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.757570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.771001 (+ 13431us) server_negotiation.cc:407] Connection header received
0810 03:19:33.771249 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.771255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.771412 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.771524 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:33.775318 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.776612 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.779684 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.780395 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.780533 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.788284 (+ 7751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:33.788291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:33.788296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:33.788342 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:33.792306 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.793644 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.793657 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.793660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.793851 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:33.795965 (+ 2114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:33.795971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:33.795974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:33.796358 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:33.796659 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.797272 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.797543 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15916,"thread_start_us":31649,"threads_started":1}
I20260810 03:19:33.803447 27169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:19:33.811640 27169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:33.857674 28403 heartbeater.cc:503] Master 127.19.246.254:34045 was elected leader, sending a full tablet report...
I20260810 03:19:33.870042 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.840120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.19.246.194:34505)
0810 03:19:33.851609 (+ 11489us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.851619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.859205 (+ 7586us) server_negotiation.cc:407] Connection header received
0810 03:19:33.859385 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.859398 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.859828 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.860098 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:33.862400 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.864319 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.867776 (+ 3457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.868933 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.869180 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.869386 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.869543 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.869717 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11262,"thread_start_us":11030,"threads_started":1}
I20260810 03:19:33.870126 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.839681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:34505 (local address 127.0.0.1:40828)
0810 03:19:33.858764 (+ 19083us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:33.858795 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:33.859008 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:33.861451 (+ 2443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:33.861455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:33.861477 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:33.862102 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.862123 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.864467 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.864475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.867342 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.867392 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.869180 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.869187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.869360 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.869393 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:19:33.869490 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 03:19:33.869652 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18848,"mutex_wait_us":101,"thread_start_us":20093,"threads_started":1}
I20260810 03:19:33.871444 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.843129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.19.246.195:38391)
0810 03:19:33.856427 (+ 13298us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.856444 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.860687 (+ 4243us) server_negotiation.cc:407] Connection header received
0810 03:19:33.860912 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.860920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.861165 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.861400 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:33.862332 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.864121 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.868063 (+ 3942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.869678 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.869863 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.869981 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.870840 (+ 859us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.871032 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13001,"thread_start_us":16063,"threads_started":1}
I20260810 03:19:33.872761 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.839459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.19.246.193:44087)
0810 03:19:33.864130 (+ 24671us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:33.864141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:33.864171 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:19:33.864283 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:33.864291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:33.864585 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:33.864816 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:33.865549 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.867267 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.870071 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.871626 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.871813 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.871984 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:33.872262 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 03:19:33.872436 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24466,"thread_start_us":25518,"threads_started":1}
I20260810 03:19:33.872807 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.838879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:44087 (local address 127.0.0.1:50844)
0810 03:19:33.849639 (+ 10760us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:33.849687 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:33.849938 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:33.864810 (+ 14872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:33.864820 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:33.864849 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:33.865372 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.865384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.867443 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.867451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.869657 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.869669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.871838 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.871844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.872066 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.872098 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:19:33.872215 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 03:19:33.872362 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10497,"mutex_wait_us":29,"thread_start_us":21343,"threads_started":1}
I20260810 03:19:33.872869 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:33.842327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:38391 (local address 127.0.0.1:44380)
0810 03:19:33.860540 (+ 18213us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:33.860559 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:33.860732 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:33.861386 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:33.861390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:33.861419 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:33.862129 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.862141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.864285 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.864291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.866970 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:33.866988 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:33.870443 (+ 3455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:33.870449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:33.870626 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:33.870659 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:19:33.870754 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:19:33.870893 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18099,"mutex_wait_us":155,"thread_start_us":6792,"threads_started":1}
I20260810 03:19:33.874620 28328 tablet_service.cc:1467] Processing CreateTablet for tablet fc6aef2b93b74e88b664a096bfecce12 (DEFAULT_TABLE table=test-table [id=03a586460b7942dcaebc8f545a0f8169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:33.876268 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6aef2b93b74e88b664a096bfecce12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:33.877103 28490 tablet_service.cc:1467] Processing CreateTablet for tablet fc6aef2b93b74e88b664a096bfecce12 (DEFAULT_TABLE table=test-table [id=03a586460b7942dcaebc8f545a0f8169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:33.878456 28490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6aef2b93b74e88b664a096bfecce12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:33.878118 28163 tablet_service.cc:1467] Processing CreateTablet for tablet fc6aef2b93b74e88b664a096bfecce12 (DEFAULT_TABLE table=test-table [id=03a586460b7942dcaebc8f545a0f8169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:33.879571 28163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6aef2b93b74e88b664a096bfecce12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:33.892627 28606 tablet_bootstrap.cc:492] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Bootstrap starting.
I20260810 03:19:33.897006 28606 tablet_bootstrap.cc:654] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:33.898448 28608 tablet_bootstrap.cc:492] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Bootstrap starting.
I20260810 03:19:33.898729 28606 log.cc:824] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:33.903107 28607 tablet_bootstrap.cc:492] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Bootstrap starting.
I20260810 03:19:33.905606 28608 tablet_bootstrap.cc:654] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:33.907729 28608 log.cc:824] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:33.910858 28606 tablet_bootstrap.cc:492] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: No bootstrap required, opened a new log
I20260810 03:19:33.911226 28606 ts_tablet_manager.cc:1397] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.010s
I20260810 03:19:33.911597 28607 tablet_bootstrap.cc:654] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:33.913134 28608 tablet_bootstrap.cc:492] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: No bootstrap required, opened a new log
I20260810 03:19:33.913095 28607 log.cc:824] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:33.913532 28608 ts_tablet_manager.cc:1397] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260810 03:19:33.924418 28608 raft_consensus.cc:348] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:33.925020 28608 raft_consensus.cc:374] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:33.925130 28608 raft_consensus.cc:729] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b316fe967e474cd9806b4cb7e30af850, State: Initialized, Role: FOLLOWER
I20260810 03:19:33.925870 28608 consensus_queue.cc:260] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:33.926083 28606 raft_consensus.cc:348] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:33.926599 28606 raft_consensus.cc:374] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:33.926733 28606 raft_consensus.cc:729] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0827a2564a94e71b9bacc423b64d5d8, State: Initialized, Role: FOLLOWER
I20260810 03:19:33.927294 28607 tablet_bootstrap.cc:492] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: No bootstrap required, opened a new log
I20260810 03:19:33.927673 28607 ts_tablet_manager.cc:1397] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.011s
I20260810 03:19:33.927654 28606 consensus_queue.cc:260] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:33.928082 28608 ts_tablet_manager.cc:1428] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260810 03:19:33.937453 28606 ts_tablet_manager.cc:1428] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Time spent starting tablet: real 0.026s user 0.021s sys 0.000s
I20260810 03:19:33.937453 28566 heartbeater.cc:503] Master 127.19.246.254:34045 was elected leader, sending a full tablet report...
I20260810 03:19:33.939323 28607 raft_consensus.cc:348] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:33.939939 28607 raft_consensus.cc:374] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:33.940066 28607 raft_consensus.cc:729] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ee6f0413794d16a015e1d1befde0a6, State: Initialized, Role: FOLLOWER
I20260810 03:19:33.940954 28607 consensus_queue.cc:260] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:33.943168 28607 ts_tablet_manager.cc:1428] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
W20260810 03:19:33.971870 28573 tablet.cc:2369] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:19:34.042446 28406 tablet.cc:2369] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:19:34.057227 28246 tablet.cc:2369] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:19:34.183926 28617 raft_consensus.cc:482] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:34.184337 28617 raft_consensus.cc:504] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:34.186314 28617 leader_election.cc:288] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087), a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391)
I20260810 03:19:34.194312 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.187014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:57575 (local address 127.19.246.193:44087)
0810 03:19:34.187496 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:34.187501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:34.188347 (+ 846us) server_negotiation.cc:407] Connection header received
0810 03:19:34.188592 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:34.188597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:34.188670 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:34.188773 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:34.189413 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.190150 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.192362 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.193258 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.193479 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.193563 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:34.193875 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 03:19:34.194016 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260810 03:19:34.194453 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.186802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:44087 (local address 127.19.246.194:57575)
0810 03:19:34.188220 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:34.188236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:34.188440 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:34.188795 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:34.188800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:34.188832 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:34.189234 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:34.189244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.190316 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.190322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:34.192163 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:34.192180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.193497 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.193503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:34.193692 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.193718 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:19:34.193832 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:19:34.194016 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":131,"spinlock_wait_cycles":15872,"thread_start_us":430,"threads_started":1}
I20260810 03:19:34.194595 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.187011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:60317 (local address 127.19.246.195:38391)
0810 03:19:34.187269 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:34.187278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:34.188342 (+ 1064us) server_negotiation.cc:407] Connection header received
0810 03:19:34.188605 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:34.188610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:34.188749 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:34.188866 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:34.189484 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.190282 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.192475 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.193384 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.193604 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.193712 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:34.194151 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 03:19:34.194306 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:19:34.195253 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.186802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:38391 (local address 127.19.246.194:60317)
0810 03:19:34.188210 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:34.188228 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:34.188439 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:34.188896 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:34.188901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:34.188928 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:34.189319 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:34.189330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.190453 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.190460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:34.192260 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:34.192274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.193661 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.193669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:34.193845 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.193872 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:19:34.194648 (+ 776us) client_negotiation.cc:240] Negotiation successful
0810 03:19:34.194825 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":22,"spinlock_wait_cycles":9600,"thread_start_us":647,"threads_started":1}
I20260810 03:19:34.196347 28185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6aef2b93b74e88b664a096bfecce12" candidate_uuid: "c3ee6f0413794d16a015e1d1befde0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b316fe967e474cd9806b4cb7e30af850" is_pre_election: true
I20260810 03:19:34.196514 28511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6aef2b93b74e88b664a096bfecce12" candidate_uuid: "c3ee6f0413794d16a015e1d1befde0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0827a2564a94e71b9bacc423b64d5d8" is_pre_election: true
I20260810 03:19:34.197109 28511 raft_consensus.cc:2454] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ee6f0413794d16a015e1d1befde0a6 in term 0.
I20260810 03:19:34.197109 28185 raft_consensus.cc:2454] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ee6f0413794d16a015e1d1befde0a6 in term 0.
I20260810 03:19:34.198211 28280 leader_election.cc:302] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a0827a2564a94e71b9bacc423b64d5d8, b316fe967e474cd9806b4cb7e30af850, c3ee6f0413794d16a015e1d1befde0a6; no voters:
I20260810 03:19:34.199054 28617 raft_consensus.cc:2781] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:34.199266 28617 raft_consensus.cc:482] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:34.199417 28617 raft_consensus.cc:3037] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:34.202132 28617 raft_consensus.cc:504] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:34.202908 28617 leader_election.cc:288] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [CANDIDATE]: Term 1 election: Requested vote from peers b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087), a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391)
I20260810 03:19:34.203603 28511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6aef2b93b74e88b664a096bfecce12" candidate_uuid: "c3ee6f0413794d16a015e1d1befde0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0827a2564a94e71b9bacc423b64d5d8"
I20260810 03:19:34.203564 28185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6aef2b93b74e88b664a096bfecce12" candidate_uuid: "c3ee6f0413794d16a015e1d1befde0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b316fe967e474cd9806b4cb7e30af850"
I20260810 03:19:34.203856 28511 raft_consensus.cc:3037] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:34.203867 28185 raft_consensus.cc:3037] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:34.206461 28185 raft_consensus.cc:2454] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ee6f0413794d16a015e1d1befde0a6 in term 1.
I20260810 03:19:34.206559 28511 raft_consensus.cc:2454] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ee6f0413794d16a015e1d1befde0a6 in term 1.
I20260810 03:19:34.207114 28278 leader_election.cc:302] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b316fe967e474cd9806b4cb7e30af850, c3ee6f0413794d16a015e1d1befde0a6; no voters:
I20260810 03:19:34.207578 28617 raft_consensus.cc:2781] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:34.208127 28617 raft_consensus.cc:686] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 1 LEADER]: Becoming Leader. State: Replica: c3ee6f0413794d16a015e1d1befde0a6, State: Running, Role: LEADER
I20260810 03:19:34.208774 28617 consensus_queue.cc:237] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } }
I20260810 03:19:34.215984 27168 catalog_manager.cc:5495] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3ee6f0413794d16a015e1d1befde0a6 (127.19.246.194). New cstate: current_term: 1 leader_uuid: "c3ee6f0413794d16a015e1d1befde0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260810 03:19:34.258796 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.250107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.19.246.194:34505)
0810 03:19:34.250344 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:34.250350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:34.250375 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:19:34.250758 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:34.250763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:34.250921 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:34.251097 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:34.251980 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.253065 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.255381 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.256033 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.256210 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.257340 (+ 1130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:34.258047 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:34.258239 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:34.258457 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 03:19:34.258554 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 03:19:34.270112 28185 raft_consensus.cc:1261] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 1 FOLLOWER]: Refusing update from remote peer c3ee6f0413794d16a015e1d1befde0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:34.270112 28511 raft_consensus.cc:1261] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 1 FOLLOWER]: Refusing update from remote peer c3ee6f0413794d16a015e1d1befde0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:34.271585 28620 consensus_queue.cc:1035] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:34.272020 28617 consensus_queue.cc:1035] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:34.287204 28623 mvcc.cc:205] Tried to move back new op lower bound from 7316815766595674112 to 7316815766364082176. Current Snapshot: MvccSnapshot[applied={T|T < 7316815766595674112}]
I20260810 03:19:34.318100 27169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:34.318681 27169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:34.329056 27169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10020f23c05d4fecaa7a383cb192d439: Sending DeleteTablet for 3 replicas of tablet fc6aef2b93b74e88b664a096bfecce12
I20260810 03:19:34.330658 28163 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6aef2b93b74e88b664a096bfecce12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:34 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:50844
I20260810 03:19:34.330893 28490 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6aef2b93b74e88b664a096bfecce12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:34 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:44380
I20260810 03:19:34.332913 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6aef2b93b74e88b664a096bfecce12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:34 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:40828
I20260810 03:19:34.344409 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 27136
I20260810 03:19:34.345759 27203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:34.352247 28634 tablet_replica.cc:330] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: stopping tablet replica
I20260810 03:19:34.352926 28634 raft_consensus.cc:2229] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:34.353783 28634 raft_consensus.cc:2258] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:34.355329 28635 tablet_replica.cc:330] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: stopping tablet replica
I20260810 03:19:34.357851 28635 raft_consensus.cc:2229] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:34.360903 28635 raft_consensus.cc:2258] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:34.363530 28636 tablet_replica.cc:330] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: stopping tablet replica
I20260810 03:19:34.364436 28636 raft_consensus.cc:2229] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:19:34.365756 28636 raft_consensus.cc:2258] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:34.378988 28636 ts_tablet_manager.cc:1905] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:34.385622 28634 ts_tablet_manager.cc:1905] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:34.393410 28635 ts_tablet_manager.cc:1905] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:34.396941 28634 ts_tablet_manager.cc:1918] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:34.397267 28634 log.cc:1196] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/fc6aef2b93b74e88b664a096bfecce12
I20260810 03:19:34.398031 28634 ts_tablet_manager.cc:1939] T fc6aef2b93b74e88b664a096bfecce12 P a0827a2564a94e71b9bacc423b64d5d8: Deleting consensus metadata
I20260810 03:19:34.410789 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.339403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.19.246.253:39327)
0810 03:19:34.340551 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:34.340562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:34.342209 (+ 1647us) server_negotiation.cc:407] Connection header received
0810 03:19:34.342316 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:34.342325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:34.342445 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:34.342593 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:34.346383 (+ 3790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.350403 (+ 4020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.383547 (+ 33144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.384738 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.384936 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.396086 (+ 11150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:34.396096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:34.396102 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:34.396156 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:34.401705 (+ 5549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:34.407025 (+ 5320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:34.407034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:34.407038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:34.407397 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:34.408114 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:34.408120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:34.408124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:34.408721 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:34.409096 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:34.409923 (+ 827us) server_negotiation.cc:299] Negotiation successful
0810 03:19:34.410247 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":714,"threads_started":1}
I20260810 03:19:34.411002 28636 ts_tablet_manager.cc:1918] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:34.411314 28636 log.cc:1196] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/fc6aef2b93b74e88b664a096bfecce12
I20260810 03:19:34.412295 28636 ts_tablet_manager.cc:1939] T fc6aef2b93b74e88b664a096bfecce12 P c3ee6f0413794d16a015e1d1befde0a6: Deleting consensus metadata
I20260810 03:19:34.418062 28635 ts_tablet_manager.cc:1918] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:34.418359 28635 log.cc:1196] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/fc6aef2b93b74e88b664a096bfecce12
I20260810 03:19:34.419320 28635 ts_tablet_manager.cc:1939] T fc6aef2b93b74e88b664a096bfecce12 P b316fe967e474cd9806b4cb7e30af850: Deleting consensus metadata
I20260810 03:19:34.440156 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.346092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.19.246.252:38643)
0810 03:19:34.347854 (+ 1762us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:34.347861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:34.413650 (+ 65789us) server_negotiation.cc:407] Connection header received
0810 03:19:34.413925 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:34.413931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:34.414020 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:34.414166 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:34.417052 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.418478 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.421311 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.422310 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.422453 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.427707 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:34.427716 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:34.427721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:34.427778 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:34.433356 (+ 5578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:34.435037 (+ 1681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:34.435044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:34.435049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:34.435445 (+ 396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:34.436488 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:34.436496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:34.436501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:34.437136 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:34.437613 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:34.438630 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0810 03:19:34.439385 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1627,"thread_start_us":1435,"threads_started":1}
I20260810 03:19:34.528570 27153 catalog_manager.cc:4841] TS b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087): tablet fc6aef2b93b74e88b664a096bfecce12 (table test-table [id=03a586460b7942dcaebc8f545a0f8169]) successfully deleted
I20260810 03:19:34.529457 27152 catalog_manager.cc:4841] TS c3ee6f0413794d16a015e1d1befde0a6 (127.19.246.194:34505): tablet fc6aef2b93b74e88b664a096bfecce12 (table test-table [id=03a586460b7942dcaebc8f545a0f8169]) successfully deleted
I20260810 03:19:34.529675 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 27136
I20260810 03:19:34.588469 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.564080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.19.246.252:38643)
0810 03:19:34.564334 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:34.564341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:34.565639 (+ 1298us) server_negotiation.cc:407] Connection header received
0810 03:19:34.565864 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:34.565875 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:34.566021 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:34.566170 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:34.571219 (+ 5049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.572152 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.574184 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.575279 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.575479 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.580727 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:34.580735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:34.580758 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:34.580821 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:34.584670 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:34.585838 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:34.585846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:34.585849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:34.586020 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:34.586476 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:34.586482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:34.586485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:34.586842 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:34.587218 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:34.587676 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 03:19:34.587914 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:19:34.591919 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:34.566956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.19.246.253:39327)
0810 03:19:34.567316 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:34.567324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:34.567996 (+ 672us) server_negotiation.cc:407] Connection header received
0810 03:19:34.568363 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:34.568370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:34.568515 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:34.568637 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:34.574379 (+ 5742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.575737 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.577634 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:34.578090 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:34.578254 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:34.582597 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:34.582605 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:34.582610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:34.582666 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:34.588180 (+ 5514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:34.589076 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:34.589081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:34.589083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:34.589248 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:34.589679 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:34.589683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:34.589685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:34.589998 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:34.590730 (+ 732us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:34.590951 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 03:19:34.591262 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 03:19:35.296859 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.295566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.194:40989)
0810 03:19:35.296383 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.296531 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":535,"threads_started":1}
W20260810 03:19:35.297351 28403 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:34045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
I20260810 03:19:35.307538 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.305544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.195:46825)
0810 03:19:35.306227 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.306818 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":393,"threads_started":1}
W20260810 03:19:35.308305 28566 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:34045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
I20260810 03:19:35.311903 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.310695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.193:38765)
0810 03:19:35.311291 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.311674 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":309,"threads_started":1}
W20260810 03:19:35.312659 28238 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:34045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
I20260810 03:19:35.828889 28651 raft_consensus.cc:482] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10020f23c05d4fecaa7a383cb192d439)
I20260810 03:19:35.829241 28651 raft_consensus.cc:504] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } }
I20260810 03:19:35.830217 28652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10020f23c05d4fecaa7a383cb192d439)
I20260810 03:19:35.830410 28652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } }
I20260810 03:19:35.831491 28652 leader_election.cc:288] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10020f23c05d4fecaa7a383cb192d439 (127.19.246.254:34045), a70cafd63bc949f2a59ec75087135282 (127.19.246.253:39327)
I20260810 03:19:35.832355 28651 leader_election.cc:288] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10020f23c05d4fecaa7a383cb192d439 (127.19.246.254:34045), 49b1c2860da94233aef20970525d2791 (127.19.246.252:38643)
I20260810 03:19:35.832986 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.830903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:59944)
0810 03:19:35.832344 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.832689 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1278,"thread_start_us":1333,"threads_started":1}
W20260810 03:19:35.833556 27811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
I20260810 03:19:35.833868 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.831565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:59954)
0810 03:19:35.832896 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.833242 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1228,"thread_start_us":1085,"threads_started":1}
W20260810 03:19:35.836462 28020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
I20260810 03:19:35.836755 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.836027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:59958)
0810 03:19:35.836459 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.836564 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
W20260810 03:19:35.837052 27811 leader_election.cc:334] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10020f23c05d4fecaa7a383cb192d439 (127.19.246.254:34045): Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
I20260810 03:19:35.839006 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.838273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:59972)
0810 03:19:35.838561 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.838787 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260810 03:19:35.839416 28020 leader_election.cc:334] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10020f23c05d4fecaa7a383cb192d439 (127.19.246.254:34045): Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
I20260810 03:19:35.841523 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.831261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.19.246.252:38643)
0810 03:19:35.832071 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:35.832080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:35.832713 (+ 633us) server_negotiation.cc:407] Connection header received
0810 03:19:35.833060 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:35.833067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:35.833153 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:35.833238 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:35.834402 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.835926 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.838729 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.840471 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.840684 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.840764 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:35.841064 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 03:19:35.841243 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":495,"threads_started":1}
I20260810 03:19:35.841518 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.830980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:38643 (local address 127.0.0.1:42922)
0810 03:19:35.832554 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.832582 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:35.832848 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:35.833491 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:35.833499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:35.833543 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:35.834017 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.834046 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.836115 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.836120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.838534 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.838551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.840715 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.840720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.840861 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.840890 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 03:19:35.840984 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:19:35.841091 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":1050,"threads_started":1}
I20260810 03:19:35.842334 28045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a70cafd63bc949f2a59ec75087135282" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49b1c2860da94233aef20970525d2791" is_pre_election: true
I20260810 03:19:35.842772 28045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a70cafd63bc949f2a59ec75087135282 in term 1.
I20260810 03:19:35.843662 27810 leader_election.cc:302] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49b1c2860da94233aef20970525d2791, a70cafd63bc949f2a59ec75087135282; no voters: 10020f23c05d4fecaa7a383cb192d439
I20260810 03:19:35.844194 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.833105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39327 (local address 127.0.0.1:50834)
0810 03:19:35.834107 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.834131 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:35.834404 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:35.835696 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:35.835704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:35.835736 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:35.836191 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.836208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.838384 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.838392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.841064 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.841091 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.842973 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.842981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.843344 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.843431 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 03:19:35.843592 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 03:19:35.843782 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":79,"thread_start_us":3039,"threads_started":1}
I20260810 03:19:35.844398 28651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 03:19:35.844635 28651 raft_consensus.cc:482] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10020f23c05d4fecaa7a383cb192d439)
I20260810 03:19:35.844795 28651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:19:35.845835 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.833470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.19.246.253:39327)
0810 03:19:35.835294 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:35.835302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:35.835322 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:19:35.835416 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:35.835421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:35.835505 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:35.836227 (+ 722us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:35.836561 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.838236 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.841291 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.842649 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.842849 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.842941 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:35.843617 (+ 676us) server_negotiation.cc:299] Negotiation successful
0810 03:19:35.843794 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1652,"thread_start_us":1392,"threads_started":1}
I20260810 03:19:35.846897 27836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49b1c2860da94233aef20970525d2791" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a70cafd63bc949f2a59ec75087135282" is_pre_election: true
I20260810 03:19:35.848429 28651 raft_consensus.cc:504] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } }
I20260810 03:19:35.849216 27836 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49b1c2860da94233aef20970525d2791 in current term 2: Already voted for candidate a70cafd63bc949f2a59ec75087135282 in this term.
I20260810 03:19:35.849349 28651 leader_election.cc:288] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [CANDIDATE]: Term 2 election: Requested vote from peers 10020f23c05d4fecaa7a383cb192d439 (127.19.246.254:34045), 49b1c2860da94233aef20970525d2791 (127.19.246.252:38643)
I20260810 03:19:35.849969 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.849487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:59978)
0810 03:19:35.849675 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.849791 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 03:19:35.850109 28045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a70cafd63bc949f2a59ec75087135282" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49b1c2860da94233aef20970525d2791"
I20260810 03:19:35.850447 28045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:19:35.850481 28020 leader_election.cc:302] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49b1c2860da94233aef20970525d2791; no voters: 10020f23c05d4fecaa7a383cb192d439, a70cafd63bc949f2a59ec75087135282
I20260810 03:19:35.851307 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.850869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:59990)
0810 03:19:35.851050 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.851159 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 03:19:35.851540 27811 leader_election.cc:334] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10020f23c05d4fecaa7a383cb192d439 (127.19.246.254:34045): Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
I20260810 03:19:35.853112 28045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a70cafd63bc949f2a59ec75087135282 in term 2.
I20260810 03:19:35.853319 28652 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:19:35.853641 27810 leader_election.cc:302] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49b1c2860da94233aef20970525d2791, a70cafd63bc949f2a59ec75087135282; no voters: 10020f23c05d4fecaa7a383cb192d439
I20260810 03:19:35.853986 28651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:19:35.854491 28651 raft_consensus.cc:686] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [term 2 LEADER]: Becoming Leader. State: Replica: a70cafd63bc949f2a59ec75087135282, State: Running, Role: LEADER
I20260810 03:19:35.854893 28651 consensus_queue.cc:237] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } }
I20260810 03:19:35.856967 28661 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a70cafd63bc949f2a59ec75087135282. Latest consensus state: current_term: 2 leader_uuid: "a70cafd63bc949f2a59ec75087135282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:35.857338 28661 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:35.858139 28664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:35.862644 28664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:35.864141 28664 catalog_manager.cc:1259] Loaded cluster ID: 62ad2ac64ac9400cb63c1b9e00918d0b
I20260810 03:19:35.864261 28664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:35.866024 28664 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:35.867344 28664 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Loaded TSK: 0
I20260810 03:19:35.868547 28664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:19:35.922365 27827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:19:35.930008 27827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:35.934840 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.934365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:60032)
0810 03:19:35.934535 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.934681 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 03:19:35.934990 28045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [term 2 FOLLOWER]: Refusing update from remote peer a70cafd63bc949f2a59ec75087135282: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 03:19:35.935873 28661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 03:19:35.936144 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.935558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:60038)
0810 03:19:35.935827 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.935925 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260810 03:19:35.936386 27811 consensus_peers.cc:487] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 -> Peer 10020f23c05d4fecaa7a383cb192d439 (127.19.246.254:34045): Couldn't send request to peer 10020f23c05d4fecaa7a383cb192d439. Status: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 03:19:35.939286 28652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a70cafd63bc949f2a59ec75087135282. Latest consensus state: current_term: 2 leader_uuid: "a70cafd63bc949f2a59ec75087135282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:35.939685 28652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:19:35.941845 28651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a70cafd63bc949f2a59ec75087135282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:35.942276 28651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:35.943711 28652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a70cafd63bc949f2a59ec75087135282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "10020f23c05d4fecaa7a383cb192d439" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34045 } } peers { permanent_uuid: "a70cafd63bc949f2a59ec75087135282" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39327 } attrs { promote: false } } peers { permanent_uuid: "49b1c2860da94233aef20970525d2791" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 38643 } attrs { promote: false } } }
I20260810 03:19:35.944021 28652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b1c2860da94233aef20970525d2791 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:19:35.973057 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.952386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:34505 (local address 127.0.0.1:56510)
0810 03:19:35.952550 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.952568 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:35.952772 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:35.966810 (+ 14038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:35.966817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:35.966856 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:35.967121 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.967134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.968869 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.968878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.970693 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.970708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.972226 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.972234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.972414 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.972454 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:19:35.972555 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:19:35.972728 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260810 03:19:35.974292 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.952505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.19.246.194:34505)
0810 03:19:35.966226 (+ 13721us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:35.966236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:35.966265 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:19:35.966363 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:35.966369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:35.966485 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:35.967272 (+ 787us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:35.967527 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.968659 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.970902 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.972023 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.973640 (+ 1617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.973739 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:35.973801 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 03:19:35.973958 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13557,"thread_start_us":14095,"threads_started":1}
I20260810 03:19:35.974813 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.953528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:44087 (local address 127.0.0.1:55722)
0810 03:19:35.966863 (+ 13335us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.966892 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:35.967007 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:35.967640 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:35.967645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:35.967673 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:35.968027 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.968039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.969384 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.969391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.971786 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.971802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.973710 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.973716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.973904 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.973938 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:19:35.974042 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 03:19:35.974190 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13192,"thread_start_us":12768,"threads_started":1}
I20260810 03:19:35.974974 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.954065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.19.246.193:44087)
0810 03:19:35.965788 (+ 11723us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:35.965796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:35.967138 (+ 1342us) server_negotiation.cc:407] Connection header received
0810 03:19:35.967236 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:35.967243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:35.967321 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:35.967448 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:35.968259 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.969228 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.972008 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.973384 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.973629 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.973734 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:35.974099 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 03:19:35.974258 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11574,"thread_start_us":11483,"threads_started":1}
I20260810 03:19:35.975247 28328 tablet_service.cc:1467] Processing CreateTablet for tablet c85276e36a2b400b8169a172ed55242c (DEFAULT_TABLE table=test-table [id=a587b8c99bdb42f890073b43d364623b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:35.975795 28328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85276e36a2b400b8169a172ed55242c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:35.976259 28163 tablet_service.cc:1467] Processing CreateTablet for tablet c85276e36a2b400b8169a172ed55242c (DEFAULT_TABLE table=test-table [id=a587b8c99bdb42f890073b43d364623b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:35.976774 28163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85276e36a2b400b8169a172ed55242c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:35.978233 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.952914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:38391 (local address 127.0.0.1:42146)
0810 03:19:35.953072 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.953092 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:35.953223 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:35.970185 (+ 16962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:35.970191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:35.970222 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:35.970682 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.970696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.972383 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.972391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.974597 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:35.974615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.977387 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.977394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:35.977591 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.977645 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 03:19:35.977751 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:19:35.977918 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":3}
I20260810 03:19:35.978935 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.953333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.19.246.195:38391)
0810 03:19:35.969796 (+ 16463us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:35.969803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:35.969823 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:19:35.969897 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:35.969901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:35.969992 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:35.970849 (+ 857us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:35.971057 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.972222 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.975439 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:35.976830 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:35.977026 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:35.977109 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:35.978446 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0810 03:19:35.978623 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16317,"thread_start_us":16173,"threads_started":1}
I20260810 03:19:35.979915 28490 tablet_service.cc:1467] Processing CreateTablet for tablet c85276e36a2b400b8169a172ed55242c (DEFAULT_TABLE table=test-table [id=a587b8c99bdb42f890073b43d364623b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:35.980680 28490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85276e36a2b400b8169a172ed55242c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:35.982271 28675 tablet_bootstrap.cc:492] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: Bootstrap starting.
I20260810 03:19:35.984157 28675 tablet_bootstrap.cc:654] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:35.986825 28675 tablet_bootstrap.cc:492] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: No bootstrap required, opened a new log
I20260810 03:19:35.987064 28675 ts_tablet_manager.cc:1397] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 03:19:35.987967 28677 tablet_bootstrap.cc:492] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: Bootstrap starting.
I20260810 03:19:35.988536 28675 raft_consensus.cc:348] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:35.988812 28675 raft_consensus.cc:374] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:35.988919 28675 raft_consensus.cc:729] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ee6f0413794d16a015e1d1befde0a6, State: Initialized, Role: FOLLOWER
I20260810 03:19:35.989233 28675 consensus_queue.cc:260] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:35.990507 28675 ts_tablet_manager.cc:1428] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260810 03:19:35.990829 28677 tablet_bootstrap.cc:654] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:35.998610 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.991445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.194:38079)
0810 03:19:35.997693 (+ 6248us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.998194 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6027,"thread_start_us":5640,"threads_started":1}
I20260810 03:19:36.000154 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:35.999642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.194:48931)
0810 03:19:35.999902 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:35.999999 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260810 03:19:36.000869 28679 tablet_bootstrap.cc:492] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: Bootstrap starting.
I20260810 03:19:36.001070 28677 tablet_bootstrap.cc:492] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: No bootstrap required, opened a new log
I20260810 03:19:36.001230 28677 ts_tablet_manager.cc:1397] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260810 03:19:36.001749 28405 heartbeater.cc:503] Master 127.19.246.253:39327 was elected leader, sending a full tablet report...
I20260810 03:19:36.002135 28677 raft_consensus.cc:348] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:36.002404 28677 raft_consensus.cc:374] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:36.002516 28677 raft_consensus.cc:729] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b316fe967e474cd9806b4cb7e30af850, State: Initialized, Role: FOLLOWER
I20260810 03:19:36.003002 28679 tablet_bootstrap.cc:654] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:36.002848 28677 consensus_queue.cc:260] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:36.003912 28677 ts_tablet_manager.cc:1428] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 03:19:36.006188 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.004580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.193:48319)
0810 03:19:36.005721 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.005903 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":824,"threads_started":1}
I20260810 03:19:36.007658 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.006942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.193:44219)
0810 03:19:36.007115 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.007489 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 03:19:36.009040 28243 heartbeater.cc:503] Master 127.19.246.253:39327 was elected leader, sending a full tablet report...
I20260810 03:19:36.009156 28679 tablet_bootstrap.cc:492] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: No bootstrap required, opened a new log
I20260810 03:19:36.009347 28679 ts_tablet_manager.cc:1397] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260810 03:19:36.010351 28679 raft_consensus.cc:348] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:36.010643 28679 raft_consensus.cc:374] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:36.010768 28679 raft_consensus.cc:729] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0827a2564a94e71b9bacc423b64d5d8, State: Initialized, Role: FOLLOWER
I20260810 03:19:36.011026 28679 consensus_queue.cc:260] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:36.012151 28679 ts_tablet_manager.cc:1428] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 03:19:36.015414 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.012850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.195:41479)
0810 03:19:36.013547 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.015050 (+ 1503us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":349,"threads_started":1}
I20260810 03:19:36.015461 28567 heartbeater.cc:503] Master 127.19.246.253:39327 was elected leader, sending a full tablet report...
I20260810 03:19:36.016912 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.016538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.195:39869)
0810 03:19:36.016716 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.016792 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 03:19:36.192492 28678 raft_consensus.cc:482] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:36.192839 28678 raft_consensus.cc:504] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:36.193610 28678 leader_election.cc:288] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391), b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087)
I20260810 03:19:36.194377 28511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85276e36a2b400b8169a172ed55242c" candidate_uuid: "c3ee6f0413794d16a015e1d1befde0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0827a2564a94e71b9bacc423b64d5d8" is_pre_election: true
I20260810 03:19:36.194397 28185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85276e36a2b400b8169a172ed55242c" candidate_uuid: "c3ee6f0413794d16a015e1d1befde0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b316fe967e474cd9806b4cb7e30af850" is_pre_election: true
I20260810 03:19:36.194676 28511 raft_consensus.cc:2454] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ee6f0413794d16a015e1d1befde0a6 in term 0.
I20260810 03:19:36.194686 28185 raft_consensus.cc:2454] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ee6f0413794d16a015e1d1befde0a6 in term 0.
I20260810 03:19:36.195266 28280 leader_election.cc:302] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a0827a2564a94e71b9bacc423b64d5d8, b316fe967e474cd9806b4cb7e30af850, c3ee6f0413794d16a015e1d1befde0a6; no voters:
I20260810 03:19:36.195644 28678 raft_consensus.cc:2781] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:36.195796 28678 raft_consensus.cc:482] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:36.195907 28678 raft_consensus.cc:3037] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:36.199215 28678 raft_consensus.cc:504] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:36.200160 28678 leader_election.cc:288] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [CANDIDATE]: Term 1 election: Requested vote from peers a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391), b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087)
I20260810 03:19:36.201025 28511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85276e36a2b400b8169a172ed55242c" candidate_uuid: "c3ee6f0413794d16a015e1d1befde0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0827a2564a94e71b9bacc423b64d5d8"
I20260810 03:19:36.201042 28185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c85276e36a2b400b8169a172ed55242c" candidate_uuid: "c3ee6f0413794d16a015e1d1befde0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b316fe967e474cd9806b4cb7e30af850"
I20260810 03:19:36.201283 28511 raft_consensus.cc:3037] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:36.201284 28185 raft_consensus.cc:3037] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:36.204149 28511 raft_consensus.cc:2454] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ee6f0413794d16a015e1d1befde0a6 in term 1.
I20260810 03:19:36.204305 28185 raft_consensus.cc:2454] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ee6f0413794d16a015e1d1befde0a6 in term 1.
I20260810 03:19:36.204813 28280 leader_election.cc:302] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0827a2564a94e71b9bacc423b64d5d8, c3ee6f0413794d16a015e1d1befde0a6; no voters:
I20260810 03:19:36.205211 28678 raft_consensus.cc:2781] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:36.205828 28678 raft_consensus.cc:686] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 1 LEADER]: Becoming Leader. State: Replica: c3ee6f0413794d16a015e1d1befde0a6, State: Running, Role: LEADER
I20260810 03:19:36.206147 28678 consensus_queue.cc:237] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } }
I20260810 03:19:36.207075 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.206564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.194:57463)
0810 03:19:36.206774 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.206886 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 03:19:36.209600 27826 catalog_manager.cc:5495] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3ee6f0413794d16a015e1d1befde0a6 (127.19.246.194). New cstate: current_term: 1 leader_uuid: "c3ee6f0413794d16a015e1d1befde0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ee6f0413794d16a015e1d1befde0a6" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 } health_report { overall_health: UNKNOWN } } }
I20260810 03:19:36.281392 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.273759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.19.246.194:34505)
0810 03:19:36.273981 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:36.273986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:36.274183 (+ 197us) server_negotiation.cc:407] Connection header received
0810 03:19:36.274325 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:36.274330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:36.274413 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:36.274530 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:36.275313 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:36.276715 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:36.279028 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:36.279751 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:36.279920 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:36.280516 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:36.280685 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:36.280796 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:36.281013 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 03:19:36.281113 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:19:36.285290 28511 raft_consensus.cc:1261] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 1 FOLLOWER]: Refusing update from remote peer c3ee6f0413794d16a015e1d1befde0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:36.285611 28185 raft_consensus.cc:1261] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 1 FOLLOWER]: Refusing update from remote peer c3ee6f0413794d16a015e1d1befde0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:36.286103 28692 consensus_queue.cc:1035] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0827a2564a94e71b9bacc423b64d5d8" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:36.286561 28678 consensus_queue.cc:1035] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b316fe967e474cd9806b4cb7e30af850" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:36.289640 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.288850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.195:40195)
0810 03:19:36.289173 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.289326 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260810 03:19:36.290551 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.289596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.193:41435)
0810 03:19:36.290036 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.290222 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260810 03:19:36.297266 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.296735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.195:36321)
0810 03:19:36.296980 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.297079 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 03:19:36.308914 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.307755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.194:37493)
0810 03:19:36.308254 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.308556 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333}
I20260810 03:19:36.308920 28698 mvcc.cc:205] Tried to move back new op lower bound from 7316815774856736768 to 7316815774543020032. Current Snapshot: MvccSnapshot[applied={T|T < 7316815774856736768}]
I20260810 03:19:36.310808 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.310198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.193:36589)
0810 03:19:36.310506 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.310612 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260810 03:19:36.311012 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.310430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.19.246.194:50227)
0810 03:19:36.310708 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.310814 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260810 03:19:36.324972 27826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:36.325344 27826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:36.332803 27826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a70cafd63bc949f2a59ec75087135282: Sending DeleteTablet for 3 replicas of tablet c85276e36a2b400b8169a172ed55242c
I20260810 03:19:36.334075 28328 tablet_service.cc:1514] Processing DeleteTablet for tablet c85276e36a2b400b8169a172ed55242c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:36 UTC) from {username='oryx', principal='oryx/127.19.246.253@KRBTEST.COM'} at 127.0.0.1:56510
I20260810 03:19:36.334619 28163 tablet_service.cc:1514] Processing DeleteTablet for tablet c85276e36a2b400b8169a172ed55242c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:36 UTC) from {username='oryx', principal='oryx/127.19.246.253@KRBTEST.COM'} at 127.0.0.1:55722
I20260810 03:19:36.334627 28490 tablet_service.cc:1514] Processing DeleteTablet for tablet c85276e36a2b400b8169a172ed55242c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:36 UTC) from {username='oryx', principal='oryx/127.19.246.253@KRBTEST.COM'} at 127.0.0.1:42146
I20260810 03:19:36.336941 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 28006
I20260810 03:19:36.349453 28710 tablet_replica.cc:330] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: stopping tablet replica
I20260810 03:19:36.349776 28710 raft_consensus.cc:2229] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:36.350229 28710 raft_consensus.cc:2258] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:36.350993 28709 tablet_replica.cc:330] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: stopping tablet replica
I20260810 03:19:36.351217 28709 raft_consensus.cc:2229] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:36.351586 28709 raft_consensus.cc:2258] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:36.352734 28709 ts_tablet_manager.cc:1905] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:36.354425 28708 tablet_replica.cc:330] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: stopping tablet replica
I20260810 03:19:36.354648 28708 raft_consensus.cc:2229] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:19:36.355094 28708 raft_consensus.cc:2258] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:36.356856 28710 ts_tablet_manager.cc:1905] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:36.359236 28708 ts_tablet_manager.cc:1905] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:36.359594 28709 ts_tablet_manager.cc:1918] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:36.359766 28709 log.cc:1196] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/c85276e36a2b400b8169a172ed55242c
I20260810 03:19:36.360307 28709 ts_tablet_manager.cc:1939] T c85276e36a2b400b8169a172ed55242c P b316fe967e474cd9806b4cb7e30af850: Deleting consensus metadata
I20260810 03:19:36.360908 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.360314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:60042)
0810 03:19:36.360617 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.360733 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260810 03:19:36.362021 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.361573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:60056)
0810 03:19:36.361801 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.361883 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 03:19:36.362715 27811 catalog_manager.cc:4841] TS b316fe967e474cd9806b4cb7e30af850 (127.19.246.193:44087): tablet c85276e36a2b400b8169a172ed55242c (table test-table [id=a587b8c99bdb42f890073b43d364623b]) successfully deleted
I20260810 03:19:36.366457 28710 ts_tablet_manager.cc:1918] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:36.366626 28710 log.cc:1196] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/c85276e36a2b400b8169a172ed55242c
I20260810 03:19:36.367092 28710 ts_tablet_manager.cc:1939] T c85276e36a2b400b8169a172ed55242c P a0827a2564a94e71b9bacc423b64d5d8: Deleting consensus metadata
I20260810 03:19:36.368454 28708 ts_tablet_manager.cc:1918] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:36.368405 27813 catalog_manager.cc:4841] TS a0827a2564a94e71b9bacc423b64d5d8 (127.19.246.195:38391): tablet c85276e36a2b400b8169a172ed55242c (table test-table [id=a587b8c99bdb42f890073b43d364623b]) successfully deleted
I20260810 03:19:36.368641 28708 log.cc:1196] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/c85276e36a2b400b8169a172ed55242c
I20260810 03:19:36.369055 28708 ts_tablet_manager.cc:1939] T c85276e36a2b400b8169a172ed55242c P c3ee6f0413794d16a015e1d1befde0a6: Deleting consensus metadata
I20260810 03:19:36.370873 27810 catalog_manager.cc:4841] TS c3ee6f0413794d16a015e1d1befde0a6 (127.19.246.194:34505): tablet c85276e36a2b400b8169a172ed55242c (table test-table [id=a587b8c99bdb42f890073b43d364623b]) successfully deleted
I20260810 03:19:36.374351 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.338425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.19.246.193:44087)
0810 03:19:36.338828 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:36.338836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:36.350980 (+ 12144us) server_negotiation.cc:407] Connection header received
0810 03:19:36.351790 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:36.351798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:36.351896 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:36.351997 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:36.354386 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:36.355828 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:36.358103 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:36.358894 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:36.359084 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:36.364812 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:36.364838 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:36.364860 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:36.364917 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:36.370067 (+ 5150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:36.371278 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:36.371290 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:36.371294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:36.371546 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:36.372395 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:36.372400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:36.372404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:36.372918 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:36.373263 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:36.373708 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 03:19:36.373980 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260810 03:19:36.375332 28232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:36.480470 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 28006
I20260810 03:19:36.508163 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 28250
I20260810 03:19:36.528124 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.509530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.19.246.194:34505)
0810 03:19:36.509882 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:36.509889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:36.509918 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:19:36.509989 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:36.509994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:36.510064 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:36.510210 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:36.512099 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:36.513055 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:36.514833 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:36.515462 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:36.515657 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:36.519822 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:36.519850 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:36.519874 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:36.519954 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:36.524497 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:36.525525 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:36.525534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:36.525536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:36.525729 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:36.526209 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:36.526213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:36.526216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:36.526698 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:36.526975 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:36.527453 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 03:19:36.527702 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260810 03:19:36.528853 28397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:36.676331 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 28250
I20260810 03:19:36.707631 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 28419
I20260810 03:19:36.729101 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.708551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.19.246.195:38391)
0810 03:19:36.709126 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:36.709133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:36.709165 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:19:36.709243 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:36.709249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:36.709335 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:36.709460 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:36.712014 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:36.713354 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:36.715134 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:36.716020 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:36.716199 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:36.720777 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:36.720800 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:36.720817 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:36.720862 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:36.725046 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:36.726456 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:36.726466 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:36.726468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:36.726729 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:36.727283 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:36.727287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:36.727296 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:36.727805 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:36.728097 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:36.728567 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 03:19:36.728825 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":315,"threads_started":1}
I20260810 03:19:36.730036 28560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:36.839756 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 28419
I20260810 03:19:36.847096 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.846582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:60060)
0810 03:19:36.846823 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.846931 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 03:19:36.848456 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:36.848017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34045 (local address 127.0.0.1:60076)
0810 03:19:36.848231 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:36.848316 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 03:19:36.867941 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 27680
I20260810 03:19:36.869851 27859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:37.036514 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 27680
I20260810 03:19:37.068174 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 27793
I20260810 03:19:37.069456 28069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:37.203521 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 27793
2026-08-10T03:19:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12308 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:19:37 dist-test-slave-btw8 krb5kdc[28720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:19:37 dist-test-slave-btw8 krb5kdc[28720](info): set up 2 sockets
Aug 10 03:19:37 dist-test-slave-btw8 krb5kdc[28720](info): commencing operation
krb5kdc: starting...
W20260810 03:19:38.524672 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.003s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:19:38 dist-test-slave-btw8 krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:19:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.254.keytab.
Entry for principal oryx/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.254.keytab.
I20260810 03:19:38.853788 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:43837
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.254.keytab
--principal=oryx/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:43837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:39.273442 28734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:39.273813 28734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:39.273988 28734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:39.282263 28734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:39.282418 28734 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:19:39.282498 28734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:39.282608 28734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:39.282722 28734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:39.282841 28734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:39.294921 28734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:43837
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.254.keytab
--principal=oryx/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:43837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:39.296443 28734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:39.297914 28734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:39.313205 28740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:39.313290 28741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:39.313174 28743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:39.315341 28734 server_base.cc:1034] running on GCE node
I20260810 03:19:39.316371 28734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:39.318452 28734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:39.319769 28734 hybrid_clock.cc:648] HybridClock initialized: now 1786331979319698 us; error 42 us; skew 500 ppm
I20260810 03:19:39.320345 28734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:39 dist-test-slave-btw8 krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331979, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:39.324646 28734 init.cc:377] Logged in from keytab as oryx/127.19.246.254@KRBTEST.COM (short username oryx)
I20260810 03:19:39.326591 28734 webserver.cc:460] Webserver started at http://127.19.246.254:33393/ using document root <none> and password file <none>
I20260810 03:19:39.327533 28734 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:39.327689 28734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:39.328107 28734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:39.331672 28734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "5634da3f9ec94ff5b3dba1cb50142952"
format_stamp: "Formatted at 2026-08-10 03:19:39 on dist-test-slave-btw8"
server_key: "ed9f89a9943c6dba455d13f294d31c55"
server_key_iv: "3de837f99e1b83cc9e113a7e9ec9dee9"
server_key_version: "encryptionkey@0"
I20260810 03:19:39.332623 28734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "5634da3f9ec94ff5b3dba1cb50142952"
format_stamp: "Formatted at 2026-08-10 03:19:39 on dist-test-slave-btw8"
server_key: "ed9f89a9943c6dba455d13f294d31c55"
server_key_iv: "3de837f99e1b83cc9e113a7e9ec9dee9"
server_key_version: "encryptionkey@0"
I20260810 03:19:39.338882 28734 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:19:39.343319 28751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:39.345137 28734 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:19:39.345438 28734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "5634da3f9ec94ff5b3dba1cb50142952"
format_stamp: "Formatted at 2026-08-10 03:19:39 on dist-test-slave-btw8"
server_key: "ed9f89a9943c6dba455d13f294d31c55"
server_key_iv: "3de837f99e1b83cc9e113a7e9ec9dee9"
server_key_version: "encryptionkey@0"
I20260810 03:19:39.345722 28734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:39.374817 28734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:39.380442 28734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:39.380870 28734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:39.408440 28734 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:43837
I20260810 03:19:39.408474 28825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:43837 every 8 connection(s)
I20260810 03:19:39.410190 28734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:39.412994 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 28734
I20260810 03:19:39.413302 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:39.414438 20443 external_mini_cluster.cc:1428] Setting key c7b5a383be1647906f7739d8bef9367f
I20260810 03:19:39.416610 28826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:19:39 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:39.434078 28826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: Bootstrap starting.
I20260810 03:19:39.438369 28826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:39.439795 28826 log.cc:824] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:39.443434 28826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: No bootstrap required, opened a new log
I20260810 03:19:39.444007 28830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:39.417455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.19.246.254:43837)
0810 03:19:39.418441 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:39.418454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:39.418499 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:19:39.420319 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:39.420346 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:39.420808 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:39.421220 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:39.423647 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:39.425397 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:39.427507 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:39.428337 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:39.428510 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:39.434076 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:39.434086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:39.434104 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:39.434160 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:39.439003 (+ 4843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:39.440004 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:39.440016 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:39.440028 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:39.440250 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:39.440748 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:39.440752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:39.440754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:39.441402 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:39.441757 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:39.442257 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 03:19:39.442757 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":508,"threads_started":1}
I20260810 03:19:39.453464 28826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5634da3f9ec94ff5b3dba1cb50142952" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43837 } }
I20260810 03:19:39.453969 28826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:39.454089 28826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5634da3f9ec94ff5b3dba1cb50142952, State: Initialized, Role: FOLLOWER
I20260810 03:19:39.454730 28826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5634da3f9ec94ff5b3dba1cb50142952" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43837 } }
I20260810 03:19:39.454999 28826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:39.455168 28826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:39.455401 28826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:39.458065 28826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5634da3f9ec94ff5b3dba1cb50142952" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43837 } }
I20260810 03:19:39.458775 28826 leader_election.cc:302] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5634da3f9ec94ff5b3dba1cb50142952; no voters:
I20260810 03:19:39.459735 28826 leader_election.cc:288] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:39.459976 28832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:39.461033 28832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [term 1 LEADER]: Becoming Leader. State: Replica: 5634da3f9ec94ff5b3dba1cb50142952, State: Running, Role: LEADER
I20260810 03:19:39.462230 28832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5634da3f9ec94ff5b3dba1cb50142952" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43837 } }
I20260810 03:19:39.463452 28826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:39.466128 28833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5634da3f9ec94ff5b3dba1cb50142952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5634da3f9ec94ff5b3dba1cb50142952" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43837 } } }
I20260810 03:19:39.466537 28834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5634da3f9ec94ff5b3dba1cb50142952. Latest consensus state: current_term: 1 leader_uuid: "5634da3f9ec94ff5b3dba1cb50142952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5634da3f9ec94ff5b3dba1cb50142952" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43837 } } }
I20260810 03:19:39.466754 28833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:39.469807 28834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:39.470070 28839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:39.477137 28839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:39.490171 28839 catalog_manager.cc:1347] Generated new cluster ID: 91817ed4f860466bbcda25a73736d8ae
I20260810 03:19:39.490334 28839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:39.513525 28839 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:39.515723 28839 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:39.533236 28839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: Generated new TSK 0
I20260810 03:19:39.534324 28839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab.
Entry for principal oryx/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab.
I20260810 03:19:39.632839 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43837
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab
--principal=oryx/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:40.052069 28856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:40.052871 28856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:40.053123 28856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:40.064968 28856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:40.065117 28856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:40.065265 28856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:19:40.065429 28856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:40.077472 28856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab
--principal=oryx/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:43837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:40.079402 28856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:40.081506 28856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:40.099970 28862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:40.100351 28861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:40.100672 28864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:40.102775 28856 server_base.cc:1034] running on GCE node
I20260810 03:19:40.103734 28856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:40.105043 28856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:40.106608 28856 hybrid_clock.cc:648] HybridClock initialized: now 1786331980106401 us; error 61 us; skew 500 ppm
I20260810 03:19:40.107475 28856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:40 dist-test-slave-btw8 krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331980, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:40.113106 28856 init.cc:377] Logged in from keytab as oryx/127.19.246.193@KRBTEST.COM (short username oryx)
I20260810 03:19:40.114992 28856 webserver.cc:460] Webserver started at http://127.19.246.193:37489/ using document root <none> and password file <none>
I20260810 03:19:40.115903 28856 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:40.116022 28856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:40.116336 28856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:40.119890 28856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "9ded42fde8c240d29dd915484f5c3725"
format_stamp: "Formatted at 2026-08-10 03:19:40 on dist-test-slave-btw8"
server_key: "5fb04b9509a8ca4b0f85f079b9e8f38b"
server_key_iv: "df1fbdaff79e1f621a61eaba37c65996"
server_key_version: "encryptionkey@0"
I20260810 03:19:40.120802 28856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "9ded42fde8c240d29dd915484f5c3725"
format_stamp: "Formatted at 2026-08-10 03:19:40 on dist-test-slave-btw8"
server_key: "5fb04b9509a8ca4b0f85f079b9e8f38b"
server_key_iv: "df1fbdaff79e1f621a61eaba37c65996"
server_key_version: "encryptionkey@0"
I20260810 03:19:40.127605 28856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260810 03:19:40.132422 28872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:40.134330 28856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:19:40.134557 28856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "9ded42fde8c240d29dd915484f5c3725"
format_stamp: "Formatted at 2026-08-10 03:19:40 on dist-test-slave-btw8"
server_key: "5fb04b9509a8ca4b0f85f079b9e8f38b"
server_key_iv: "df1fbdaff79e1f621a61eaba37c65996"
server_key_version: "encryptionkey@0"
I20260810 03:19:40.134835 28856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:40.199697 28856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:40.205175 28856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:40.205488 28856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:40.208783 28856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:40.208901 28856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:40.209071 28856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:40.209152 28856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:19:40 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331980, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:40.241552 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:40.209886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:55983 (local address 127.19.246.254:43837)
0810 03:19:40.210492 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:40.210498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:40.212377 (+ 1879us) server_negotiation.cc:407] Connection header received
0810 03:19:40.215596 (+ 3219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:40.215605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:40.215744 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:40.215837 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:40.219098 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:40.220626 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:40.224027 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:40.224719 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:40.224942 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:40.233441 (+ 8499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:40.233447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:40.233450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:40.233495 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:40.236892 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:40.238319 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:40.238325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:40.238327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:40.238551 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:40.239146 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:40.239153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:40.239159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:40.239594 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:40.239785 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:40.240841 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0810 03:19:40.241155 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":346,"threads_started":1}
I20260810 03:19:40.242709 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:40.210873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43837 (local address 127.19.246.193:55983)
0810 03:19:40.212138 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:40.212196 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:40.215083 (+ 2887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:40.216558 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:40.216567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:40.217589 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:40.218710 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:40.218773 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:40.220966 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:40.220977 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:40.223749 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:40.223773 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:40.225201 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:40.225216 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:40.225745 (+ 529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:40.227213 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:40.227262 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:40.233057 (+ 5795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:40.237113 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:40.237126 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:40.237156 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:40.238068 (+ 912us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:40.238706 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:40.238712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:40.238715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:40.238994 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:40.239754 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:40.239763 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:40.240371 (+ 608us) client_negotiation.cc:769] Sending connection context
0810 03:19:40.240711 (+ 340us) client_negotiation.cc:240] Negotiation successful
0810 03:19:40.241338 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":599,"threads_started":1}
I20260810 03:19:40.272969 28856 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:41259
I20260810 03:19:40.273048 29008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:41259 every 8 connection(s)
I20260810 03:19:40.274858 28856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:40.276269 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:40.253797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:46271 (local address 127.19.246.254:43837)
0810 03:19:40.254158 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:40.254168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:40.256611 (+ 2443us) server_negotiation.cc:407] Connection header received
0810 03:19:40.256880 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:40.256891 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:40.257032 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:40.257216 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:40.259283 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:40.260862 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:40.262807 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:40.263851 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:40.264137 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:40.267412 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:40.267424 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:40.267433 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:40.267516 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:40.272215 (+ 4699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:40.273646 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:40.273651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:40.273654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:40.273816 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:40.274337 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:40.274341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:40.274343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:40.274728 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:40.274875 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:40.275574 (+ 699us) server_negotiation.cc:299] Negotiation successful
0810 03:19:40.275882 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 03:19:40.276826 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:40.253812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43837 (local address 127.19.246.193:46271)
0810 03:19:40.256403 (+ 2591us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:40.256429 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:40.256667 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:40.257272 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:40.257280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:40.258179 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:40.258905 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:40.258928 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:40.261181 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:40.261193 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:40.262564 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:40.262581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:40.264341 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:40.264357 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:40.264658 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:40.265624 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:40.265663 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:40.267022 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:40.272536 (+ 5514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:40.272543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:40.272549 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:40.273494 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:40.273977 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:40.273984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:40.273988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:40.274181 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:40.274901 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:40.274907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:40.275155 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 03:19:40.275413 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 03:19:40.275962 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2410,"thread_start_us":2264,"threads_started":1}
I20260810 03:19:40.278719 29009 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43837
I20260810 03:19:40.279237 29009 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:40.280055 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 28856
I20260810 03:19:40.280237 29009 heartbeater.cc:511] Master 127.19.246.254:43837 requested a full tablet report, sending...
I20260810 03:19:40.280313 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:19:40.281402 20443 external_mini_cluster.cc:1428] Setting key 759a61bf2382e06125afda5393c2d9a1
I20260810 03:19:40.283774 28770 ts_manager.cc:194] Registered new tserver with Master: 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259)
I20260810 03:19:40.285626 28770 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:55983
WARNING: no policy specified for oryx/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab.
Entry for principal oryx/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab.
I20260810 03:19:40.356587 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43837
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab
--principal=oryx/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:40.847146 29014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:40.847776 29014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:40.848034 29014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:40.861644 29014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:40.861961 29014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:40.862201 29014 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:19:40.862476 29014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:40.883046 29014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab
--principal=oryx/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:43837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:40.886240 29014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:40.890173 29014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:40.919698 29019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:40.919564 29022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:40.919744 29020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:40.921475 29014 server_base.cc:1034] running on GCE node
I20260810 03:19:40.922988 29014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:40.925575 29014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:40.927253 29014 hybrid_clock.cc:648] HybridClock initialized: now 1786331980927076 us; error 54 us; skew 500 ppm
I20260810 03:19:40.928516 29014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:40 dist-test-slave-btw8 krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331980, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:40.937327 29014 init.cc:377] Logged in from keytab as oryx/127.19.246.194@KRBTEST.COM (short username oryx)
I20260810 03:19:40.940762 29014 webserver.cc:460] Webserver started at http://127.19.246.194:44717/ using document root <none> and password file <none>
I20260810 03:19:40.942162 29014 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:40.942421 29014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:40.943023 29014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:40.950176 29014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "475e306b3f074a04b92e3cbb90e94bae"
format_stamp: "Formatted at 2026-08-10 03:19:40 on dist-test-slave-btw8"
server_key: "d1b240d615f2e1bcb78acb6ceb172d81"
server_key_iv: "5936fa0c4f8484abc440005ec4c9a013"
server_key_version: "encryptionkey@0"
I20260810 03:19:40.954955 29014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "475e306b3f074a04b92e3cbb90e94bae"
format_stamp: "Formatted at 2026-08-10 03:19:40 on dist-test-slave-btw8"
server_key: "d1b240d615f2e1bcb78acb6ceb172d81"
server_key_iv: "5936fa0c4f8484abc440005ec4c9a013"
server_key_version: "encryptionkey@0"
I20260810 03:19:40.979319 29014 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.013s sys 0.005s
I20260810 03:19:40.985157 29029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:40.987672 29014 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:19:40.987987 29014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "475e306b3f074a04b92e3cbb90e94bae"
format_stamp: "Formatted at 2026-08-10 03:19:40 on dist-test-slave-btw8"
server_key: "d1b240d615f2e1bcb78acb6ceb172d81"
server_key_iv: "5936fa0c4f8484abc440005ec4c9a013"
server_key_version: "encryptionkey@0"
I20260810 03:19:40.988435 29014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:41.007629 29014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:41.014869 29014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:41.015256 29014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:41.019663 29014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:41.019806 29014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:41.019980 29014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:41.020072 29014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:19:41 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331980, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:41.052254 29039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:41.020274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:34503 (local address 127.19.246.254:43837)
0810 03:19:41.021146 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:41.021191 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:41.022386 (+ 1195us) server_negotiation.cc:407] Connection header received
0810 03:19:41.025453 (+ 3067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:41.025460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:41.025590 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:41.025739 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:41.028757 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:41.030587 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:41.034356 (+ 3769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:41.035200 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:41.035430 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:41.042766 (+ 7336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:41.042777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:41.042784 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:41.042850 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:41.047412 (+ 4562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:41.048589 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:41.048596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:41.048599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:41.048844 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:41.049445 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:41.049451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:41.049454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:41.049813 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:41.050025 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:41.051428 (+ 1403us) server_negotiation.cc:299] Negotiation successful
0810 03:19:41.051795 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":626,"threads_started":1}
I20260810 03:19:41.054229 29040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:41.021003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43837 (local address 127.19.246.194:34503)
0810 03:19:41.022153 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:41.022209 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:41.025042 (+ 2833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:41.026085 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:41.026102 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:41.027013 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:41.028391 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:41.028439 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:41.030915 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:41.030929 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:41.034031 (+ 3102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:41.034069 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:41.035715 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:41.035723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:41.035976 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:41.037111 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:41.037175 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:41.042286 (+ 5111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:41.047718 (+ 5432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:41.047741 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:41.047760 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:41.048382 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:41.049004 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:41.049019 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:41.049022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:41.049288 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:41.049992 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:41.050003 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:41.050548 (+ 545us) client_negotiation.cc:769] Sending connection context
0810 03:19:41.051094 (+ 546us) client_negotiation.cc:240] Negotiation successful
0810 03:19:41.052013 (+ 919us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":443,"threads_started":1}
I20260810 03:19:41.083695 29039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:41.063795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:50639 (local address 127.19.246.254:43837)
0810 03:19:41.064054 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:41.064062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:41.065199 (+ 1137us) server_negotiation.cc:407] Connection header received
0810 03:19:41.065427 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:41.065437 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:41.065567 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:41.065721 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:41.067118 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:41.068528 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:41.071062 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:41.071962 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:41.072173 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:41.075763 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:41.075769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:41.075773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:41.075818 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:41.080115 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:41.081308 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:41.081316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:41.081320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:41.081482 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:41.081920 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:41.081926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:41.081929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:41.082330 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:41.082499 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:41.082992 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 03:19:41.083283 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 03:19:41.083997 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:41.063826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43837 (local address 127.19.246.194:50639)
0810 03:19:41.064990 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:41.065022 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:41.065261 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:41.065813 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:41.065817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:41.066516 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:41.066910 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:41.066922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:41.068759 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:41.068767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:41.070801 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:41.070830 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:41.072321 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:41.072330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:41.072634 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:41.073748 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:41.073787 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:41.075328 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:41.080385 (+ 5057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:41.080394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:41.080398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:41.081112 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:41.081626 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:41.081631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:41.081635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:41.081791 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:41.082484 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:41.082489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:41.082720 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 03:19:41.082882 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 03:19:41.083285 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":725,"threads_started":1}
I20260810 03:19:41.095423 29014 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:39893
I20260810 03:19:41.095456 29161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:39893 every 8 connection(s)
I20260810 03:19:41.097231 29014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:41.100031 29162 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43837
I20260810 03:19:41.100297 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 29014
I20260810 03:19:41.100423 29162 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:41.100500 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:19:41.101121 20443 external_mini_cluster.cc:1428] Setting key fb986afc3fd8cb969da0e146c13d07ab
I20260810 03:19:41.101439 29162 heartbeater.cc:511] Master 127.19.246.254:43837 requested a full tablet report, sending...
I20260810 03:19:41.103295 28773 ts_manager.cc:194] Registered new tserver with Master: 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:41.105803 28773 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:34503
WARNING: no policy specified for oryx/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab.
Entry for principal oryx/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab.
W20260810 03:19:41.216868 20443 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.19.246.195: real 0.110s user 0.002s sys 0.040s
I20260810 03:19:41.218660 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43837
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab
--principal=oryx/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:41.290469 29009 heartbeater.cc:503] Master 127.19.246.254:43837 was elected leader, sending a full tablet report...
W20260810 03:19:41.891543 29166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:41.892217 29166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:41.892469 29166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:41.913213 29166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:41.913498 29166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:41.913836 29166 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:19:41.914189 29166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:41.930397 29166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab
--principal=oryx/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.246.254:43837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:41.932032 29166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:41.933686 29166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:41.962265 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:41.962417 29172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:41.962302 29174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:41.965344 29166 server_base.cc:1034] running on GCE node
I20260810 03:19:41.966665 29166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:41.968782 29166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:41.970299 29166 hybrid_clock.cc:648] HybridClock initialized: now 1786331981970150 us; error 41 us; skew 500 ppm
I20260810 03:19:41.970966 29166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:41 dist-test-slave-btw8 krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331981, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:41.976920 29166 init.cc:377] Logged in from keytab as oryx/127.19.246.195@KRBTEST.COM (short username oryx)
I20260810 03:19:41.979905 29166 webserver.cc:460] Webserver started at http://127.19.246.195:44959/ using document root <none> and password file <none>
I20260810 03:19:41.981241 29166 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:41.981405 29166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:41.981854 29166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:41.987706 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "a34c00258bcd4542b0d3b32117a3c32f"
format_stamp: "Formatted at 2026-08-10 03:19:41 on dist-test-slave-btw8"
server_key: "d68a07946e1e3e1a7f1c44e3e16dd563"
server_key_iv: "2f7fd8b2cb60a3516b8d71fbe4625a8f"
server_key_version: "encryptionkey@0"
I20260810 03:19:41.989362 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "a34c00258bcd4542b0d3b32117a3c32f"
format_stamp: "Formatted at 2026-08-10 03:19:41 on dist-test-slave-btw8"
server_key: "d68a07946e1e3e1a7f1c44e3e16dd563"
server_key_iv: "2f7fd8b2cb60a3516b8d71fbe4625a8f"
server_key_version: "encryptionkey@0"
I20260810 03:19:42.000715 29166 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.009s sys 0.002s
I20260810 03:19:42.008328 29181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:42.012228 29166 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260810 03:19:42.012672 29166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "a34c00258bcd4542b0d3b32117a3c32f"
format_stamp: "Formatted at 2026-08-10 03:19:41 on dist-test-slave-btw8"
server_key: "d68a07946e1e3e1a7f1c44e3e16dd563"
server_key_iv: "2f7fd8b2cb60a3516b8d71fbe4625a8f"
server_key_version: "encryptionkey@0"
I20260810 03:19:42.013134 29166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:42.043656 29166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:42.053812 29166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:42.054406 29166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:42.060511 29166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:42.060899 29166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:42.061187 29166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:42.061352 29166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:19:42 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331981, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:42.114843 29162 heartbeater.cc:503] Master 127.19.246.254:43837 was elected leader, sending a full tablet report...
I20260810 03:19:42.121162 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.061798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:51715 (local address 127.19.246.254:43837)
0810 03:19:42.063478 (+ 1680us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.063489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.065304 (+ 1815us) server_negotiation.cc:407] Connection header received
0810 03:19:42.072162 (+ 6858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.072169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.072581 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.072855 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:42.079643 (+ 6788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.081014 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.089164 (+ 8150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.090012 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.090323 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.102429 (+ 12106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:42.102442 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:42.102450 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:42.102557 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:42.111175 (+ 8618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:42.113903 (+ 2728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:42.113925 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:42.113932 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:42.114301 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:42.115881 (+ 1580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:42.115903 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:42.115908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:42.116831 (+ 923us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:42.119935 (+ 3104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.120313 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.120648 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1396,"thread_start_us":1220,"threads_started":1}
I20260810 03:19:42.124835 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.062950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43837 (local address 127.19.246.195:51715)
0810 03:19:42.064857 (+ 1907us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.064971 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.071055 (+ 6084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.073490 (+ 2435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.073505 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.075715 (+ 2210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:42.078438 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.078831 (+ 393us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.081480 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.081505 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.088621 (+ 7116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.088685 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.090531 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.090546 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.091844 (+ 1298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.094283 (+ 2439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:42.094404 (+ 121us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:42.101696 (+ 7292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:42.111813 (+ 10117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:42.111835 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:42.111902 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:42.113475 (+ 1573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:42.114592 (+ 1117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:42.114612 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:42.114624 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:42.115269 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:42.117223 (+ 1954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:42.117234 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:42.118687 (+ 1453us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.119934 (+ 1247us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.121090 (+ 1156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1285,"thread_start_us":662,"threads_started":1}
I20260810 03:19:42.177835 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.144682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:53415 (local address 127.19.246.254:43837)
0810 03:19:42.145052 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.145060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.146536 (+ 1476us) server_negotiation.cc:407] Connection header received
0810 03:19:42.146965 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.146971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.147075 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.147264 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:42.150901 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.152787 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.156339 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.157148 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.157396 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.162911 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:42.162932 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:42.162954 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:42.163040 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:42.169306 (+ 6266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:42.172802 (+ 3496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:42.172815 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:42.172834 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:42.173220 (+ 386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:42.174261 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:42.174269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:42.174283 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:42.174899 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:42.175415 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.176428 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.177052 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 03:19:42.178336 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.144958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43837 (local address 127.19.246.195:53415)
0810 03:19:42.146274 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.146327 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.146677 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.147334 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.147340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.148321 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:42.149867 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.149923 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.153059 (+ 3136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.153078 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.155971 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.156022 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.157545 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.157568 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.157855 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.160082 (+ 2227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:42.160107 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:42.162381 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:42.169682 (+ 7301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:42.169690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:42.169705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:42.171804 (+ 2099us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:42.173618 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:42.173630 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:42.173640 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:42.173994 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:42.175163 (+ 1169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:42.175254 (+ 91us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:42.175730 (+ 476us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.176199 (+ 469us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.176946 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":662,"threads_started":1}
I20260810 03:19:42.180593 29166 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:45863
I20260810 03:19:42.180696 29314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:45863 every 8 connection(s)
I20260810 03:19:42.183835 29166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:42.186815 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 29166
I20260810 03:19:42.187448 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:19:42.188794 20443 external_mini_cluster.cc:1428] Setting key fca02dbe4434143055366ec9cb47ff49
I20260810 03:19:42.190656 29315 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43837
I20260810 03:19:42.191309 29315 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:42.193385 29315 heartbeater.cc:511] Master 127.19.246.254:43837 requested a full tablet report, sending...
I20260810 03:19:42.196853 28773 ts_manager.cc:194] Registered new tserver with Master: a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195:45863)
I20260810 03:19:42.199606 28773 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:51715
I20260810 03:19:42.213577 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:19:42 dist-test-slave-btw8 krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:19:42 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:42 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:19:42.332158 29191 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:42.303586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.19.246.254:43837)
0810 03:19:42.304208 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.304234 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.305162 (+ 928us) server_negotiation.cc:407] Connection header received
0810 03:19:42.305680 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.305687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.305774 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.305895 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:42.309438 (+ 3543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.310676 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.313499 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.314328 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.314584 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.331544 (+ 16960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43858: BlockingRecv error: recv got EOF from 127.0.0.1:43858 (error 108)
Metrics: {"server-negotiator.queue_time_us":194}
Aug 10 03:19:42 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:42.369999 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.338331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.19.246.254:43837)
0810 03:19:42.338796 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.338820 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.339793 (+ 973us) server_negotiation.cc:407] Connection header received
0810 03:19:42.340253 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.340260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.340356 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.340547 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:42.342942 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.344382 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.346954 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.347874 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.348222 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.357488 (+ 9266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:42.357508 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:42.357513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:42.357680 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:42.363572 (+ 5892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:42.365777 (+ 2205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:42.365803 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:42.365805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:42.366111 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:42.366944 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:42.366951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:42.366954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:42.367599 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:42.368156 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.368773 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.369321 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260810 03:19:42.380245 28773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:19:42.393639 28773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:42.465042 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.444708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.19.246.194:39893)
0810 03:19:42.447658 (+ 2950us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.447682 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.448776 (+ 1094us) server_negotiation.cc:407] Connection header received
0810 03:19:42.448998 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.449008 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.449450 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.449847 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:42.451583 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.454014 (+ 2431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.458562 (+ 4548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.461515 (+ 2953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.461774 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.462190 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.463939 (+ 1749us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.464323 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2584,"thread_start_us":2295,"threads_started":1}
I20260810 03:19:42.465878 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.445006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:39893 (local address 127.0.0.1:50202)
0810 03:19:42.448266 (+ 3260us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.448295 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.448582 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.450271 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.450276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.450307 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:42.451118 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.451149 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.454367 (+ 3218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.454379 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.458211 (+ 3832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.458245 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.462620 (+ 4375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.462634 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.463198 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.463404 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.463752 (+ 348us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.464268 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3020,"mutex_wait_us":1,"spinlock_wait_cycles":2432,"thread_start_us":2685,"threads_started":1}
I20260810 03:19:42.466403 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.420689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:41259 (local address 127.0.0.1:60512)
0810 03:19:42.444416 (+ 23727us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.444433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.444573 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.448131 (+ 3558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.448143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.448237 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:42.449284 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.449313 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.452937 (+ 3624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.452944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.457627 (+ 4683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.457659 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.464711 (+ 7052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.464735 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.465121 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.465209 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.465374 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.465638 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23571,"thread_start_us":21703,"threads_started":1}
I20260810 03:19:42.466459 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.421925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.19.246.193:41259)
0810 03:19:42.441452 (+ 19527us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.441485 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.444544 (+ 3059us) server_negotiation.cc:407] Connection header received
0810 03:19:42.444738 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.444749 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.445090 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.445503 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:42.449809 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.452578 (+ 2769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.458109 (+ 5531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.461515 (+ 3406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.461774 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.462190 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.465608 (+ 3418us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.465952 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19127,"thread_start_us":19548,"threads_started":1}
I20260810 03:19:42.468029 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.420187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:45863 (local address 127.0.0.1:60140)
0810 03:19:42.443331 (+ 23144us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.443458 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.443748 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.452046 (+ 8298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.452053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.452092 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:42.452945 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.452988 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.457204 (+ 4216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.457212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.461350 (+ 4138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.461393 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.466412 (+ 5019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.466423 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.466787 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.466904 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.467034 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.467416 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22260,"thread_start_us":24238,"threads_started":1}
I20260810 03:19:42.469415 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.420621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.19.246.195:45863)
0810 03:19:42.447076 (+ 26455us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.447100 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.447219 (+ 119us) server_negotiation.cc:407] Connection header received
0810 03:19:42.448238 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.448262 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.451495 (+ 3233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.453549 (+ 2054us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:42.454103 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.456801 (+ 2698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.462773 (+ 5972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.465951 (+ 3178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.466942 (+ 991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.467410 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.467537 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.468960 (+ 1423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26045,"thread_start_us":25670,"threads_started":1}
I20260810 03:19:42.476034 29085 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3925ebb734c0298aff6160a97a8a1 (DEFAULT_TABLE table=test-table [id=3823afbd57704bff9a354eba4619a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:42.476032 28930 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3925ebb734c0298aff6160a97a8a1 (DEFAULT_TABLE table=test-table [id=3823afbd57704bff9a354eba4619a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:42.477067 29240 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3925ebb734c0298aff6160a97a8a1 (DEFAULT_TABLE table=test-table [id=3823afbd57704bff9a354eba4619a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:42.479751 28930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d3925ebb734c0298aff6160a97a8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:42.479812 29085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d3925ebb734c0298aff6160a97a8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:42.479883 29240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d3925ebb734c0298aff6160a97a8a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:42.498080 29343 tablet_bootstrap.cc:492] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Bootstrap starting.
I20260810 03:19:42.500088 29345 tablet_bootstrap.cc:492] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Bootstrap starting.
I20260810 03:19:42.501505 29342 tablet_bootstrap.cc:492] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Bootstrap starting.
I20260810 03:19:42.504516 29342 tablet_bootstrap.cc:654] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:42.504580 29345 tablet_bootstrap.cc:654] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:42.504662 29343 tablet_bootstrap.cc:654] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:42.505767 29342 log.cc:824] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:42.505957 29343 log.cc:824] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:42.506201 29345 log.cc:824] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:42.509390 29342 tablet_bootstrap.cc:492] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: No bootstrap required, opened a new log
I20260810 03:19:42.509652 29342 ts_tablet_manager.cc:1397] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 03:19:42.509753 29345 tablet_bootstrap.cc:492] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: No bootstrap required, opened a new log
I20260810 03:19:42.509768 29343 tablet_bootstrap.cc:492] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: No bootstrap required, opened a new log
I20260810 03:19:42.510042 29345 ts_tablet_manager.cc:1397] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260810 03:19:42.510048 29343 ts_tablet_manager.cc:1397] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 03:19:42.520929 29342 raft_consensus.cc:348] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.521414 29342 raft_consensus.cc:374] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:42.521553 29342 raft_consensus.cc:729] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475e306b3f074a04b92e3cbb90e94bae, State: Initialized, Role: FOLLOWER
I20260810 03:19:42.522356 29342 consensus_queue.cc:260] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.522797 29345 raft_consensus.cc:348] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.523341 29345 raft_consensus.cc:374] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:42.523507 29345 raft_consensus.cc:729] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Initialized, Role: FOLLOWER
I20260810 03:19:42.523444 29343 raft_consensus.cc:348] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.523921 29343 raft_consensus.cc:374] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:42.524021 29343 raft_consensus.cc:729] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ded42fde8c240d29dd915484f5c3725, State: Initialized, Role: FOLLOWER
I20260810 03:19:42.524462 29342 ts_tablet_manager.cc:1428] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260810 03:19:42.524282 29345 consensus_queue.cc:260] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.524832 29343 consensus_queue.cc:260] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.525895 29315 heartbeater.cc:503] Master 127.19.246.254:43837 was elected leader, sending a full tablet report...
I20260810 03:19:42.526872 29345 ts_tablet_manager.cc:1428] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260810 03:19:42.528925 29343 ts_tablet_manager.cc:1428] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
W20260810 03:19:42.530206 29011 tablet.cc:2369] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:19:42.601068 29163 tablet.cc:2369] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:19:42.612669 29350 raft_consensus.cc:482] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:42.613095 29350 raft_consensus.cc:504] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.615036 29350 leader_election.cc:288] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259), 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:42.621496 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.615638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:53205 (local address 127.19.246.193:41259)
0810 03:19:42.615879 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.615886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.615911 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:19:42.615992 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.615997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.616120 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.616223 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:42.616726 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.617608 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.619599 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.620501 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.620693 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.620779 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.621072 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.621215 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:19:42.621513 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.615325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:41259 (local address 127.19.246.195:53205)
0810 03:19:42.615593 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.615677 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.615863 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.616333 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.616340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.616370 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:42.616558 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.616567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.617836 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.617844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.619392 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.619409 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.620673 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.620679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.620897 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.620922 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.621037 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.621215 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 03:19:42.622377 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.615821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:39893 (local address 127.19.246.195:41399)
0810 03:19:42.616596 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.616623 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.616798 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.617133 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.617137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.617160 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:42.617495 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.617504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.618507 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.618514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.620144 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.620157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.621487 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.621493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.621694 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.621726 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.621857 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.621997 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":427,"threads_started":1}
I20260810 03:19:42.622526 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.615971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:41399 (local address 127.19.246.194:39893)
0810 03:19:42.616226 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.616232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.616733 (+ 501us) server_negotiation.cc:407] Connection header received
0810 03:19:42.616945 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.616949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.617009 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.617116 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:42.617676 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.618366 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.620315 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.621272 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.621460 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.621534 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.621939 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.622273 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:19:42.623564 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3925ebb734c0298aff6160a97a8a1" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae" is_pre_election: true
I20260810 03:19:42.623579 28957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3925ebb734c0298aff6160a97a8a1" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ded42fde8c240d29dd915484f5c3725" is_pre_election: true
I20260810 03:19:42.624100 29107 raft_consensus.cc:2454] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 0.
I20260810 03:19:42.624102 28957 raft_consensus.cc:2454] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 0.
I20260810 03:19:42.624785 29184 leader_election.cc:302] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475e306b3f074a04b92e3cbb90e94bae, a34c00258bcd4542b0d3b32117a3c32f; no voters:
I20260810 03:19:42.625325 29350 raft_consensus.cc:2781] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:42.625521 29350 raft_consensus.cc:482] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:42.625675 29350 raft_consensus.cc:3037] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:42.628428 29350 raft_consensus.cc:504] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.629360 29350 leader_election.cc:288] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 election: Requested vote from peers 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259), 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:42.629899 28957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3925ebb734c0298aff6160a97a8a1" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ded42fde8c240d29dd915484f5c3725"
I20260810 03:19:42.630010 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3925ebb734c0298aff6160a97a8a1" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae"
I20260810 03:19:42.630173 28957 raft_consensus.cc:3037] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:42.630216 29107 raft_consensus.cc:3037] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:42.633909 29107 raft_consensus.cc:2454] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 1.
I20260810 03:19:42.633909 28957 raft_consensus.cc:2454] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 1.
I20260810 03:19:42.634496 29184 leader_election.cc:302] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475e306b3f074a04b92e3cbb90e94bae, a34c00258bcd4542b0d3b32117a3c32f; no voters:
I20260810 03:19:42.634905 29350 raft_consensus.cc:2781] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:42.635594 29350 raft_consensus.cc:686] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 1 LEADER]: Becoming Leader. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Running, Role: LEADER
I20260810 03:19:42.636149 29350 consensus_queue.cc:237] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } }
I20260810 03:19:42.641451 28769 catalog_manager.cc:5495] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f reported cstate change: term changed from 0 to 1, leader changed from <none> to a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195). New cstate: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } health_report { overall_health: UNKNOWN } } }
W20260810 03:19:42.691509 29316 tablet.cc:2369] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:19:42.692606 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.686155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.19.246.195:45863)
0810 03:19:42.686396 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.686401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.686552 (+ 151us) server_negotiation.cc:407] Connection header received
0810 03:19:42.686806 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.686810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.686910 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.687055 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:42.687733 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.688433 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.690011 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.690478 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.690589 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.691124 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:42.691951 (+ 827us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:42.692063 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.692318 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.692396 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 03:19:42.703243 29107 raft_consensus.cc:1261] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Refusing update from remote peer a34c00258bcd4542b0d3b32117a3c32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:42.703243 28957 raft_consensus.cc:1261] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Refusing update from remote peer a34c00258bcd4542b0d3b32117a3c32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:42.704667 29350 consensus_queue.cc:1035] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:42.705102 29353 consensus_queue.cc:1035] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:42.744387 28769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:42.744757 28769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:42.749927 28769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: Sending DeleteTablet for 3 replicas of tablet 44d3925ebb734c0298aff6160a97a8a1
I20260810 03:19:42.751688 29240 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d3925ebb734c0298aff6160a97a8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:42 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60140
I20260810 03:19:42.752748 29085 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d3925ebb734c0298aff6160a97a8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:42 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:50202
I20260810 03:19:42.753051 28769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:19:42.753888 28930 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d3925ebb734c0298aff6160a97a8a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:42 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60512
I20260810 03:19:42.754194 29368 tablet_replica.cc:330] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: stopping tablet replica
I20260810 03:19:42.754841 29368 raft_consensus.cc:2229] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:42.755663 29368 raft_consensus.cc:2258] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 03:19:42.756157 28769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:42.758800 29368 ts_tablet_manager.cc:1905] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:42.763257 29367 tablet_replica.cc:330] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: stopping tablet replica
I20260810 03:19:42.763818 29367 raft_consensus.cc:2229] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:19:42.764650 29367 raft_consensus.cc:2258] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:42.767086 29367 ts_tablet_manager.cc:1905] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:42.768738 29368 ts_tablet_manager.cc:1918] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:42.769053 29368 log.cc:1196] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/44d3925ebb734c0298aff6160a97a8a1
I20260810 03:19:42.770138 29368 ts_tablet_manager.cc:1939] T 44d3925ebb734c0298aff6160a97a8a1 P 475e306b3f074a04b92e3cbb90e94bae: Deleting consensus metadata
I20260810 03:19:42.771324 29240 tablet_service.cc:1467] Processing CreateTablet for tablet 94943b3c81234b799593fc56d167afcd (DEFAULT_TABLE table=test-table [id=cfad7a0dd4004a07bae84be5eff7682c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:42.771588 28929 tablet_service.cc:1467] Processing CreateTablet for tablet 94943b3c81234b799593fc56d167afcd (DEFAULT_TABLE table=test-table [id=cfad7a0dd4004a07bae84be5eff7682c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:42.771879 29240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94943b3c81234b799593fc56d167afcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:42.772467 28929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94943b3c81234b799593fc56d167afcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:42.773399 28754 catalog_manager.cc:4841] TS 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893): tablet 44d3925ebb734c0298aff6160a97a8a1 (table test-table [id=3823afbd57704bff9a354eba4619a604]) successfully deleted
I20260810 03:19:42.775957 29085 tablet_service.cc:1467] Processing CreateTablet for tablet 94943b3c81234b799593fc56d167afcd (DEFAULT_TABLE table=test-table [id=cfad7a0dd4004a07bae84be5eff7682c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:42.776684 29085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94943b3c81234b799593fc56d167afcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:42.778209 29345 tablet_bootstrap.cc:492] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: Bootstrap starting.
I20260810 03:19:42.780252 29345 tablet_bootstrap.cc:654] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:42.783959 29367 ts_tablet_manager.cc:1918] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:42.784235 29367 log.cc:1196] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/44d3925ebb734c0298aff6160a97a8a1
I20260810 03:19:42.784938 29367 ts_tablet_manager.cc:1939] T 44d3925ebb734c0298aff6160a97a8a1 P a34c00258bcd4542b0d3b32117a3c32f: Deleting consensus metadata
I20260810 03:19:42.786718 29342 tablet_bootstrap.cc:492] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: Bootstrap starting.
I20260810 03:19:42.787402 28753 catalog_manager.cc:4841] TS a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195:45863): tablet 44d3925ebb734c0298aff6160a97a8a1 (table test-table [id=3823afbd57704bff9a354eba4619a604]) successfully deleted
I20260810 03:19:42.788702 29342 tablet_bootstrap.cc:654] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:42.789968 29369 tablet_replica.cc:330] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: stopping tablet replica
I20260810 03:19:42.790750 29369 raft_consensus.cc:2229] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:42.791680 29369 raft_consensus.cc:2258] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:42.794605 29369 ts_tablet_manager.cc:1905] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:42.796552 29343 tablet_bootstrap.cc:492] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: Bootstrap starting.
I20260810 03:19:42.799049 29343 tablet_bootstrap.cc:654] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:42.803761 29342 tablet_bootstrap.cc:492] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: No bootstrap required, opened a new log
I20260810 03:19:42.805791 29342 ts_tablet_manager.cc:1397] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.009s
I20260810 03:19:42.806728 29369 ts_tablet_manager.cc:1918] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:42.807122 29369 log.cc:1196] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/44d3925ebb734c0298aff6160a97a8a1
I20260810 03:19:42.807219 29342 raft_consensus.cc:348] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.807626 29342 raft_consensus.cc:374] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:42.807792 29342 raft_consensus.cc:729] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475e306b3f074a04b92e3cbb90e94bae, State: Initialized, Role: FOLLOWER
I20260810 03:19:42.808265 29369 ts_tablet_manager.cc:1939] T 44d3925ebb734c0298aff6160a97a8a1 P 9ded42fde8c240d29dd915484f5c3725: Deleting consensus metadata
I20260810 03:19:42.808295 29342 consensus_queue.cc:260] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.810624 29342 ts_tablet_manager.cc:1428] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 03:19:42.811102 28754 catalog_manager.cc:4841] TS 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259): tablet 44d3925ebb734c0298aff6160a97a8a1 (table test-table [id=3823afbd57704bff9a354eba4619a604]) successfully deleted
I20260810 03:19:42.814397 29343 tablet_bootstrap.cc:492] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: No bootstrap required, opened a new log
I20260810 03:19:42.814992 29343 ts_tablet_manager.cc:1397] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.010s
I20260810 03:19:42.816875 29343 raft_consensus.cc:348] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.817289 29343 raft_consensus.cc:374] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:42.817449 29343 raft_consensus.cc:729] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ded42fde8c240d29dd915484f5c3725, State: Initialized, Role: FOLLOWER
I20260810 03:19:42.817780 29343 consensus_queue.cc:260] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.818766 29343 ts_tablet_manager.cc:1428] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 03:19:42.821450 29345 tablet_bootstrap.cc:492] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: No bootstrap required, opened a new log
I20260810 03:19:42.821827 29345 ts_tablet_manager.cc:1397] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.024s
I20260810 03:19:42.823557 29345 raft_consensus.cc:348] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.824070 29345 raft_consensus.cc:374] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:42.824266 29345 raft_consensus.cc:729] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Initialized, Role: FOLLOWER
I20260810 03:19:42.824584 29345 consensus_queue.cc:260] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.826120 29345 ts_tablet_manager.cc:1428] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 03:19:42.944049 29351 raft_consensus.cc:482] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:42.944397 29351 raft_consensus.cc:504] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.946787 29351 leader_election.cc:288] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893), a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195:45863)
I20260810 03:19:42.956568 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.947472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:42001 (local address 127.19.246.194:39893)
0810 03:19:42.947701 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.947709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.948780 (+ 1071us) server_negotiation.cc:407] Connection header received
0810 03:19:42.949237 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.949244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.949344 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.949472 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:42.950547 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.951845 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.954155 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.955406 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.955647 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.955742 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.956098 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.956252 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:19:42.956727 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.947307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:39893 (local address 127.19.246.193:42001)
0810 03:19:42.948605 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.948640 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.948970 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.949692 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.949699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.949765 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:42.950317 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.950332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.952071 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.952075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.953927 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.953944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.955683 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.955690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.955908 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.955934 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.956059 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.956226 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":741,"threads_started":1}
I20260810 03:19:42.957655 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94943b3c81234b799593fc56d167afcd" candidate_uuid: "9ded42fde8c240d29dd915484f5c3725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae" is_pre_election: true
I20260810 03:19:42.957969 29107 raft_consensus.cc:2454] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ded42fde8c240d29dd915484f5c3725 in term 0.
I20260810 03:19:42.958166 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.947509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:58365 (local address 127.19.246.195:45863)
0810 03:19:42.947781 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:42.947792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:42.949021 (+ 1229us) server_negotiation.cc:407] Connection header received
0810 03:19:42.949293 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:42.949298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:42.949364 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:42.949482 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:19:42.950190 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.951687 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.954832 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.956476 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.957048 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.957163 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:42.957649 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 03:19:42.957785 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:19:42.958715 28875 leader_election.cc:302] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475e306b3f074a04b92e3cbb90e94bae, 9ded42fde8c240d29dd915484f5c3725; no voters:
I20260810 03:19:42.959147 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:42.947917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:45863 (local address 127.19.246.193:58365)
0810 03:19:42.948906 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:42.948931 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:42.949150 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:42.949515 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:42.949520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:42.949551 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:19:42.949978 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.949990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.951898 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.951906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.954556 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:42.954577 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:42.957081 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:42.957091 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:42.957357 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:42.957409 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 03:19:42.958258 (+ 849us) client_negotiation.cc:240] Negotiation successful
0810 03:19:42.958507 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"spinlock_wait_cycles":31104,"thread_start_us":652,"threads_started":1}
I20260810 03:19:42.959255 29351 raft_consensus.cc:2781] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:42.959427 29351 raft_consensus.cc:482] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:42.959527 29351 raft_consensus.cc:3037] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:42.962188 29351 raft_consensus.cc:504] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.963752 29351 leader_election.cc:288] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [CANDIDATE]: Term 1 election: Requested vote from peers 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893), a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195:45863)
I20260810 03:19:42.963524 29261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94943b3c81234b799593fc56d167afcd" candidate_uuid: "9ded42fde8c240d29dd915484f5c3725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34c00258bcd4542b0d3b32117a3c32f" is_pre_election: true
I20260810 03:19:42.963865 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94943b3c81234b799593fc56d167afcd" candidate_uuid: "9ded42fde8c240d29dd915484f5c3725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae"
I20260810 03:19:42.964146 29107 raft_consensus.cc:3037] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:42.964188 29261 raft_consensus.cc:2454] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ded42fde8c240d29dd915484f5c3725 in term 0.
I20260810 03:19:42.964954 29260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94943b3c81234b799593fc56d167afcd" candidate_uuid: "9ded42fde8c240d29dd915484f5c3725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34c00258bcd4542b0d3b32117a3c32f"
I20260810 03:19:42.965333 29260 raft_consensus.cc:3037] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:42.967799 29107 raft_consensus.cc:2454] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ded42fde8c240d29dd915484f5c3725 in term 1.
I20260810 03:19:42.968570 29260 raft_consensus.cc:2454] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ded42fde8c240d29dd915484f5c3725 in term 1.
I20260810 03:19:42.968561 28875 leader_election.cc:302] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475e306b3f074a04b92e3cbb90e94bae, 9ded42fde8c240d29dd915484f5c3725; no voters:
I20260810 03:19:42.969249 29351 raft_consensus.cc:2781] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:42.980507 29351 raft_consensus.cc:686] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 1 LEADER]: Becoming Leader. State: Replica: 9ded42fde8c240d29dd915484f5c3725, State: Running, Role: LEADER
I20260810 03:19:42.981073 29351 consensus_queue.cc:237] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:42.985870 28769 catalog_manager.cc:5495] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193). New cstate: current_term: 1 leader_uuid: "9ded42fde8c240d29dd915484f5c3725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:19:43 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331979, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.254@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:43.076776 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.054098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43837 (local address 127.0.0.1:43876)
0810 03:19:43.055313 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:43.055333 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:43.055555 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:43.056165 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:43.056171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:43.057324 (+ 1153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:43.057687 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:43.057703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.059522 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.059528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:43.060968 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:43.060989 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.062470 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.062487 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:43.062788 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.063861 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:43.063890 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:43.066651 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:43.073245 (+ 6594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:43.073265 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:43.073273 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:43.073952 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:43.074535 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:43.074539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:43.074541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:43.074708 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:43.075628 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:43.075639 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:43.075973 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 03:19:43.076151 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 03:19:43.076454 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":857,"threads_started":1}
I20260810 03:19:43.077423 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.054318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.19.246.254:43837)
0810 03:19:43.055069 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:43.055079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:43.055509 (+ 430us) server_negotiation.cc:407] Connection header received
0810 03:19:43.055688 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:43.055694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:43.055781 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:43.055926 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:43.057902 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.059230 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.061184 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.061885 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.062242 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.066969 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:43.066977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:43.066982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:43.067025 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:43.072682 (+ 5657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:43.074146 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:43.074153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:43.074157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:43.074361 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:43.074851 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:43.074857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:43.074861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:43.075269 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:43.075489 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:43.076313 (+ 824us) server_negotiation.cc:299] Negotiation successful
0810 03:19:43.076689 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":533,"thread_start_us":333,"threads_started":1}
I20260810 03:19:43.084127 28769 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:43876:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:19:43.085642 28769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:43.094523 28929 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3316aed6be4081aaab207bee0546ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e2d97c3332c490eab91f7b879b570aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:19:43.094640 29085 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3316aed6be4081aaab207bee0546ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e2d97c3332c490eab91f7b879b570aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:19:43.095227 28929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3316aed6be4081aaab207bee0546ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:43.095247 29085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3316aed6be4081aaab207bee0546ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:43.101148 29240 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3316aed6be4081aaab207bee0546ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e2d97c3332c490eab91f7b879b570aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:19:43.101639 29342 tablet_bootstrap.cc:492] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: Bootstrap starting.
I20260810 03:19:43.101929 29240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3316aed6be4081aaab207bee0546ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:43.103621 29342 tablet_bootstrap.cc:654] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:43.114427 29342 tablet_bootstrap.cc:492] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: No bootstrap required, opened a new log
I20260810 03:19:43.114688 29342 ts_tablet_manager.cc:1397] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260810 03:19:43.115535 29342 raft_consensus.cc:348] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.115833 29342 raft_consensus.cc:374] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:43.115931 29342 raft_consensus.cc:729] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475e306b3f074a04b92e3cbb90e94bae, State: Initialized, Role: FOLLOWER
I20260810 03:19:43.116219 29343 tablet_bootstrap.cc:492] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: Bootstrap starting.
I20260810 03:19:43.116215 29342 consensus_queue.cc:260] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.116895 29342 ts_tablet_manager.cc:1428] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:19:43.116762 29349 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.117383 29349 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:19:43.118933 29343 tablet_bootstrap.cc:654] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:43.125924 29345 tablet_bootstrap.cc:492] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: Bootstrap starting.
I20260810 03:19:43.128189 29345 tablet_bootstrap.cc:654] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:43.132184 29343 tablet_bootstrap.cc:492] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: No bootstrap required, opened a new log
I20260810 03:19:43.132431 29343 ts_tablet_manager.cc:1397] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.009s
I20260810 03:19:43.133582 29343 raft_consensus.cc:348] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.133906 29343 raft_consensus.cc:374] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:43.134008 29343 raft_consensus.cc:729] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ded42fde8c240d29dd915484f5c3725, State: Initialized, Role: FOLLOWER
I20260810 03:19:43.134375 29343 consensus_queue.cc:260] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.135387 29351 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.135886 29343 ts_tablet_manager.cc:1428] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260810 03:19:43.135957 29351 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:19:43.141119 29345 tablet_bootstrap.cc:492] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: No bootstrap required, opened a new log
I20260810 03:19:43.141491 29345 ts_tablet_manager.cc:1397] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20260810 03:19:43.142408 29345 raft_consensus.cc:348] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.142791 29345 raft_consensus.cc:374] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:43.142920 29345 raft_consensus.cc:729] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Initialized, Role: FOLLOWER
I20260810 03:19:43.143260 29345 consensus_queue.cc:260] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.144022 29345 ts_tablet_manager.cc:1428] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:19:43.143981 29362 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.144366 29362 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:19:43.260942 29362 raft_consensus.cc:482] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:43.261317 29362 raft_consensus.cc:504] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.262090 29362 leader_election.cc:288] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259), 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:43.262948 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3316aed6be4081aaab207bee0546ab" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae" is_pre_election: true
I20260810 03:19:43.263000 28957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3316aed6be4081aaab207bee0546ab" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ded42fde8c240d29dd915484f5c3725" is_pre_election: true
I20260810 03:19:43.263276 29107 raft_consensus.cc:2454] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 0.
I20260810 03:19:43.263399 28957 raft_consensus.cc:2454] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 0.
I20260810 03:19:43.264057 29184 leader_election.cc:302] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475e306b3f074a04b92e3cbb90e94bae, a34c00258bcd4542b0d3b32117a3c32f; no voters:
I20260810 03:19:43.264905 29362 raft_consensus.cc:2781] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:43.265118 29362 raft_consensus.cc:482] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:43.265262 29362 raft_consensus.cc:3037] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:43.268600 29362 raft_consensus.cc:504] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.269384 29362 leader_election.cc:288] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 election: Requested vote from peers 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259), 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:43.270961 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3316aed6be4081aaab207bee0546ab" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae"
I20260810 03:19:43.270733 28957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3316aed6be4081aaab207bee0546ab" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ded42fde8c240d29dd915484f5c3725"
I20260810 03:19:43.271341 29107 raft_consensus.cc:3037] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:43.271375 28957 raft_consensus.cc:3037] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:43.274906 29107 raft_consensus.cc:2454] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 1.
I20260810 03:19:43.274974 28957 raft_consensus.cc:2454] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 1.
I20260810 03:19:43.275693 29184 leader_election.cc:302] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475e306b3f074a04b92e3cbb90e94bae, a34c00258bcd4542b0d3b32117a3c32f; no voters:
I20260810 03:19:43.276531 29362 raft_consensus.cc:2781] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:43.277436 29362 raft_consensus.cc:686] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [term 1 LEADER]: Becoming Leader. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Running, Role: LEADER
I20260810 03:19:43.278082 29362 consensus_queue.cc:237] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.280305 29391 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: TxnStatusTablet state changed. Reason: New leader a34c00258bcd4542b0d3b32117a3c32f. Latest consensus state: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.280783 29391 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:19:43.281772 29393 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:19:43.282131 29393 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:19:43.283597 28769 catalog_manager.cc:5495] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f reported cstate change: term changed from 0 to 1, leader changed from <none> to a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195). New cstate: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } health_report { overall_health: HEALTHY } } }
I20260810 03:19:43.331990 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.324945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.19.246.195:45863)
0810 03:19:43.325164 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:43.325169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:43.325593 (+ 424us) server_negotiation.cc:407] Connection header received
0810 03:19:43.325917 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:43.325921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:43.325993 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:43.326090 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:43.326618 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.327424 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.329415 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.330098 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.330227 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.331145 (+ 918us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:43.331286 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:43.331438 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:43.331656 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 03:19:43.331747 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:19:43.332110 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.324968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:45863 (local address 127.0.0.1:60162)
0810 03:19:43.325438 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:43.325461 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:43.325672 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:43.326157 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:43.326162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:43.326186 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:19:43.326403 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:43.326415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.327579 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.327585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:43.329229 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:43.329248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.330527 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.330543 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:43.330818 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.330918 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:19:43.331458 (+ 540us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:19:43.331488 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:19:43.331588 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 03:19:43.331777 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260810 03:19:43.336947 29107 raft_consensus.cc:1261] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Refusing update from remote peer a34c00258bcd4542b0d3b32117a3c32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:43.337002 28957 raft_consensus.cc:1261] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Refusing update from remote peer a34c00258bcd4542b0d3b32117a3c32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:43.337697 29362 consensus_queue.cc:1035] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:43.338152 29391 consensus_queue.cc:1035] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:43.344399 29351 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: TxnStatusTablet state changed. Reason: New leader a34c00258bcd4542b0d3b32117a3c32f. Latest consensus state: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.344746 29351 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:19:43.345773 29349 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: TxnStatusTablet state changed. Reason: New leader a34c00258bcd4542b0d3b32117a3c32f. Latest consensus state: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.346143 29349 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:19:43.346951 29397 mvcc.cc:205] Tried to move back new op lower bound from 7316815803740397568 to 7316815803510280192. Current Snapshot: MvccSnapshot[applied={T|T < 7316815803740397568}]
I20260810 03:19:43.347848 29398 mvcc.cc:205] Tried to move back new op lower bound from 7316815803740397568 to 7316815803510280192. Current Snapshot: MvccSnapshot[applied={T|T < 7316815803740397568}]
I20260810 03:19:43.348546 29362 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.348875 29362 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:19:43.356699 29391 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.357038 29391 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P a34c00258bcd4542b0d3b32117a3c32f: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:19:43.359596 29399 mvcc.cc:205] Tried to move back new op lower bound from 7316815803740397568 to 7316815803510280192. Current Snapshot: MvccSnapshot[applied={T|T < 7316815803740397568}]
I20260810 03:19:43.367144 29351 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.367722 29351 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P 9ded42fde8c240d29dd915484f5c3725: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:19:43.367980 29351 consensus_queue.cc:1035] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:43.368358 29349 tablet_replica.cc:439] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } } }
I20260810 03:19:43.368642 29349 tablet_replica.cc:442] T 3b3316aed6be4081aaab207bee0546ab P 475e306b3f074a04b92e3cbb90e94bae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:19:43.374250 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.357611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.19.246.193:41259)
0810 03:19:43.358711 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:43.358720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:43.358742 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:19:43.358865 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:43.358871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:43.359314 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:43.360769 (+ 1455us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:43.361003 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.362081 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.364424 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.365170 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.365301 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.369153 (+ 3852us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:43.372148 (+ 2995us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:43.372929 (+ 781us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:43.373404 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 03:19:43.373696 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":2758,"threads_started":1}
I20260810 03:19:43.391371 29375 consensus_queue.cc:1035] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:43.410435 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.402703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:45211 (local address 127.19.246.195:45863)
0810 03:19:43.402893 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:43.402901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:43.403032 (+ 131us) server_negotiation.cc:407] Connection header received
0810 03:19:43.403267 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:43.403273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:43.403413 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:43.403536 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:43.404289 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.405784 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.407795 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.408602 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.408805 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.409501 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:43.409692 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:43.409811 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:43.409966 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:19:43.410066 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:19:43.410544 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.402192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:45863 (local address 127.19.246.193:45211)
0810 03:19:43.402891 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:43.402912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:43.403085 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:43.403582 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:43.403588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:43.403602 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:19:43.403940 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:43.403950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.406017 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.406022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:43.407592 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:43.407612 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.408869 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.408879 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:43.409166 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.409265 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:19:43.409822 (+ 557us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:19:43.409834 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:19:43.409917 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 03:19:43.410093 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":723,"threads_started":1}
I20260810 03:19:43.475452 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.466234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:34747 (local address 127.19.246.193:41259)
0810 03:19:43.466469 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:43.466479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:43.467245 (+ 766us) server_negotiation.cc:407] Connection header received
0810 03:19:43.467493 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:43.467502 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:43.467618 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:43.467823 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:43.468631 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.469765 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.472699 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.473243 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.473691 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.474415 (+ 724us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:43.474600 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:43.474675 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:43.474952 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 03:19:43.475068 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:19:43.476572 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.466226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:41259 (local address 127.19.246.195:34747)
0810 03:19:43.467096 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:43.467119 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:43.467293 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:43.467859 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:43.467865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:43.467880 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:19:43.468335 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:43.468349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.469930 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.469937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:43.472194 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:43.472218 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.473735 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.473752 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:43.474003 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.474115 (+ 112us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:19:43.474787 (+ 672us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:19:43.474801 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 03:19:43.475774 (+ 973us) client_negotiation.cc:240] Negotiation successful
0810 03:19:43.475961 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":434,"threads_started":1}
I20260810 03:19:43.534312 28773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:43.534695 28773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:43.539608 28773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: Sending DeleteTablet for 3 replicas of tablet 94943b3c81234b799593fc56d167afcd
I20260810 03:19:43.540843 29085 tablet_service.cc:1514] Processing DeleteTablet for tablet 94943b3c81234b799593fc56d167afcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:43 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:50202
I20260810 03:19:43.540855 28929 tablet_service.cc:1514] Processing DeleteTablet for tablet 94943b3c81234b799593fc56d167afcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:43 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60512
I20260810 03:19:43.541644 29240 tablet_service.cc:1514] Processing DeleteTablet for tablet 94943b3c81234b799593fc56d167afcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:43 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60140
I20260810 03:19:43.551308 29425 tablet_replica.cc:330] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: stopping tablet replica
I20260810 03:19:43.551826 29425 raft_consensus.cc:2229] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:19:43.552737 29425 raft_consensus.cc:2258] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:43.554333 29425 ts_tablet_manager.cc:1905] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:43.564009 29426 tablet_replica.cc:330] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: stopping tablet replica
I20260810 03:19:43.564499 29426 raft_consensus.cc:2229] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:43.564947 29426 raft_consensus.cc:2258] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:43.572521 29424 tablet_replica.cc:330] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: stopping tablet replica
I20260810 03:19:43.572839 29424 raft_consensus.cc:2229] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:43.573127 29424 raft_consensus.cc:2258] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:43.574586 29426 ts_tablet_manager.cc:1905] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:43.575997 29425 ts_tablet_manager.cc:1918] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:19:43.576195 29425 log.cc:1196] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/94943b3c81234b799593fc56d167afcd
I20260810 03:19:43.576766 29425 ts_tablet_manager.cc:1939] T 94943b3c81234b799593fc56d167afcd P 9ded42fde8c240d29dd915484f5c3725: Deleting consensus metadata
I20260810 03:19:43.578852 28754 catalog_manager.cc:4841] TS 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259): tablet 94943b3c81234b799593fc56d167afcd (table test-table [id=cfad7a0dd4004a07bae84be5eff7682c]) successfully deleted
I20260810 03:19:43.582039 29426 ts_tablet_manager.cc:1918] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:19:43.582433 29426 log.cc:1196] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/94943b3c81234b799593fc56d167afcd
I20260810 03:19:43.583170 29426 ts_tablet_manager.cc:1939] T 94943b3c81234b799593fc56d167afcd P a34c00258bcd4542b0d3b32117a3c32f: Deleting consensus metadata
I20260810 03:19:43.584690 29424 ts_tablet_manager.cc:1905] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:43.593492 28753 catalog_manager.cc:4841] TS a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195:45863): tablet 94943b3c81234b799593fc56d167afcd (table test-table [id=cfad7a0dd4004a07bae84be5eff7682c]) successfully deleted
I20260810 03:19:43.597360 29424 ts_tablet_manager.cc:1918] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:19:43.597805 29424 log.cc:1196] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/94943b3c81234b799593fc56d167afcd
I20260810 03:19:43.598606 29424 ts_tablet_manager.cc:1939] T 94943b3c81234b799593fc56d167afcd P 475e306b3f074a04b92e3cbb90e94bae: Deleting consensus metadata
I20260810 03:19:43.600797 28754 catalog_manager.cc:4841] TS 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893): tablet 94943b3c81234b799593fc56d167afcd (table test-table [id=cfad7a0dd4004a07bae84be5eff7682c]) successfully deleted
Aug 10 03:19:43 dist-test-slave-btw8 krb5kdc[28720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:19:43 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:43 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:19:43.675992 29433 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:43.645294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.19.246.254:43837)
0810 03:19:43.656736 (+ 11442us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:43.656743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:43.656766 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:19:43.656840 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:43.656847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:43.656941 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:43.657081 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:43.658902 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.660124 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.662368 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.662860 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.663032 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.675715 (+ 12683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43880: BlockingRecv error: recv got EOF from 127.0.0.1:43880 (error 108)
Metrics: {"server-negotiator.queue_time_us":11281,"thread_start_us":11464,"threads_started":1}
Aug 10 03:19:43 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM
I20260810 03:19:43.696628 29433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.678159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.19.246.254:43837)
0810 03:19:43.678394 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:43.678404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:43.680169 (+ 1765us) server_negotiation.cc:407] Connection header received
0810 03:19:43.680410 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:43.680415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:43.680487 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:43.680604 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:43.682160 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.682952 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.684483 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.684957 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.685122 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:43.689672 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:43.689678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:43.689682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:43.689732 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:43.692850 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:43.693912 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:43.693917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:43.693920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:43.694154 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:43.694705 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:43.694710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:43.694714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:43.695094 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:43.695453 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:43.695960 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 03:19:43.696242 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:19:43.701295 28773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:19:43.702226 28773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:43.711594 28929 tablet_service.cc:1467] Processing CreateTablet for tablet 77ff391e67f14afd8609e842d33a1518 (DEFAULT_TABLE table=test-table [id=e9faaa3f3305499892bd2c969f22b71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:43.711580 29085 tablet_service.cc:1467] Processing CreateTablet for tablet 77ff391e67f14afd8609e842d33a1518 (DEFAULT_TABLE table=test-table [id=e9faaa3f3305499892bd2c969f22b71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:43.712110 29085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ff391e67f14afd8609e842d33a1518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:43.713639 29240 tablet_service.cc:1467] Processing CreateTablet for tablet 77ff391e67f14afd8609e842d33a1518 (DEFAULT_TABLE table=test-table [id=e9faaa3f3305499892bd2c969f22b71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:43.714426 29240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ff391e67f14afd8609e842d33a1518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:43.720014 28929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ff391e67f14afd8609e842d33a1518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:43.741109 29442 tablet_bootstrap.cc:492] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: Bootstrap starting.
I20260810 03:19:43.748090 29442 tablet_bootstrap.cc:654] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:43.748276 29441 tablet_bootstrap.cc:492] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: Bootstrap starting.
I20260810 03:19:43.750372 29441 tablet_bootstrap.cc:654] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:43.754283 29443 tablet_bootstrap.cc:492] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: Bootstrap starting.
I20260810 03:19:43.758073 29443 tablet_bootstrap.cc:654] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:43.761577 29442 tablet_bootstrap.cc:492] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: No bootstrap required, opened a new log
I20260810 03:19:43.761890 29442 ts_tablet_manager.cc:1397] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.015s
I20260810 03:19:43.763182 29442 raft_consensus.cc:348] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.763893 29442 raft_consensus.cc:374] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:43.764097 29442 raft_consensus.cc:729] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ded42fde8c240d29dd915484f5c3725, State: Initialized, Role: FOLLOWER
I20260810 03:19:43.764511 29442 consensus_queue.cc:260] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.765108 29441 tablet_bootstrap.cc:492] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: No bootstrap required, opened a new log
I20260810 03:19:43.765307 29441 ts_tablet_manager.cc:1397] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20260810 03:19:43.765621 29442 ts_tablet_manager.cc:1428] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:43.766203 29441 raft_consensus.cc:348] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.766510 29441 raft_consensus.cc:374] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:43.766618 29441 raft_consensus.cc:729] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475e306b3f074a04b92e3cbb90e94bae, State: Initialized, Role: FOLLOWER
I20260810 03:19:43.766860 29441 consensus_queue.cc:260] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.767473 29441 ts_tablet_manager.cc:1428] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 03:19:43.768615 29443 tablet_bootstrap.cc:492] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: No bootstrap required, opened a new log
I20260810 03:19:43.768814 29443 ts_tablet_manager.cc:1397] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s
I20260810 03:19:43.773681 29443 raft_consensus.cc:348] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.773993 29443 raft_consensus.cc:374] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:43.774127 29443 raft_consensus.cc:729] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Initialized, Role: FOLLOWER
I20260810 03:19:43.774417 29443 consensus_queue.cc:260] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.775605 29443 ts_tablet_manager.cc:1428] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260810 03:19:43.896908 29421 raft_consensus.cc:482] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:43.897317 29421 raft_consensus.cc:504] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.908280 28957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ff391e67f14afd8609e842d33a1518" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ded42fde8c240d29dd915484f5c3725" is_pre_election: true
I20260810 03:19:43.908674 28957 raft_consensus.cc:2454] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 0.
I20260810 03:19:43.909605 29184 leader_election.cc:302] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ded42fde8c240d29dd915484f5c3725, a34c00258bcd4542b0d3b32117a3c32f; no voters:
I20260810 03:19:43.910089 29362 raft_consensus.cc:2781] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:43.910244 29362 raft_consensus.cc:482] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:43.910423 29362 raft_consensus.cc:3037] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:43.899847 29421 leader_election.cc:288] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259), 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:43.914636 29362 raft_consensus.cc:504] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.920805 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ff391e67f14afd8609e842d33a1518" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae" is_pre_election: true
I20260810 03:19:43.921182 29107 raft_consensus.cc:2454] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 0.
I20260810 03:19:43.923002 28957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ff391e67f14afd8609e842d33a1518" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ded42fde8c240d29dd915484f5c3725"
I20260810 03:19:43.923336 28957 raft_consensus.cc:3037] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:43.930611 28957 raft_consensus.cc:2454] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 1.
I20260810 03:19:43.931859 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ff391e67f14afd8609e842d33a1518" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae"
I20260810 03:19:43.932196 29107 raft_consensus.cc:3037] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:43.936058 29107 raft_consensus.cc:2454] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 1.
I20260810 03:19:43.936854 29184 leader_election.cc:302] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475e306b3f074a04b92e3cbb90e94bae, a34c00258bcd4542b0d3b32117a3c32f; no voters:
I20260810 03:19:43.937321 29362 leader_election.cc:288] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 election: Requested vote from peers 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259), 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:43.937321 29421 raft_consensus.cc:2781] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:43.937753 29421 raft_consensus.cc:686] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 1 LEADER]: Becoming Leader. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Running, Role: LEADER
I20260810 03:19:43.938129 29421 consensus_queue.cc:237] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:43.942665 28769 catalog_manager.cc:5495] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f reported cstate change: term changed from 0 to 1, leader changed from <none> to a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195). New cstate: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } health_report { overall_health: HEALTHY } } }
I20260810 03:19:44.001296 29449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:43.994050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.19.246.195:45863)
0810 03:19:43.994752 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:43.994757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:43.994773 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:19:43.994837 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:43.994840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:43.994904 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:43.995024 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:43.995478 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.996244 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.998203 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:43.998826 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:43.999183 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:44.000043 (+ 860us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:44.000209 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:44.000564 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:44.000797 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 03:19:44.000927 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":394,"threads_started":1}
I20260810 03:19:44.005637 29107 raft_consensus.cc:1261] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Refusing update from remote peer a34c00258bcd4542b0d3b32117a3c32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:44.005723 28957 raft_consensus.cc:1261] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Refusing update from remote peer a34c00258bcd4542b0d3b32117a3c32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:44.006516 29362 consensus_queue.cc:1035] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:44.006960 29421 consensus_queue.cc:1035] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:44.028079 28770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:44.028409 28770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:44.032678 28770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: Sending DeleteTablet for 3 replicas of tablet 77ff391e67f14afd8609e842d33a1518
I20260810 03:19:44.033846 28929 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ff391e67f14afd8609e842d33a1518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:44 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60512
I20260810 03:19:44.033957 29240 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ff391e67f14afd8609e842d33a1518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:44 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60140
I20260810 03:19:44.033957 29085 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ff391e67f14afd8609e842d33a1518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:44 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:50202
I20260810 03:19:44.034235 29425 tablet_replica.cc:330] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: stopping tablet replica
I20260810 03:19:44.034283 29424 tablet_replica.cc:330] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: stopping tablet replica
I20260810 03:19:44.034483 29424 raft_consensus.cc:2229] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:44.034718 29426 tablet_replica.cc:330] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: stopping tablet replica
I20260810 03:19:44.034745 29424 raft_consensus.cc:2258] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:44.034948 29426 raft_consensus.cc:2229] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:19:44.035308 29426 raft_consensus.cc:2258] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:44.036309 29425 raft_consensus.cc:2229] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:44.036659 29425 raft_consensus.cc:2258] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:44.036680 28770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:19:44.037011 29424 ts_tablet_manager.cc:1905] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 03:19:44.037473 28770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:19:44.038352 29426 ts_tablet_manager.cc:1905] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:44.042120 29425 ts_tablet_manager.cc:1905] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:44.043632 29424 ts_tablet_manager.cc:1918] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:44.043835 29424 log.cc:1196] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/77ff391e67f14afd8609e842d33a1518
I20260810 03:19:44.044350 29424 ts_tablet_manager.cc:1939] T 77ff391e67f14afd8609e842d33a1518 P 475e306b3f074a04b92e3cbb90e94bae: Deleting consensus metadata
I20260810 03:19:44.045755 28754 catalog_manager.cc:4841] TS 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893): tablet 77ff391e67f14afd8609e842d33a1518 (table test-table [id=e9faaa3f3305499892bd2c969f22b71a]) successfully deleted
I20260810 03:19:44.051832 29426 ts_tablet_manager.cc:1918] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:44.052230 29426 log.cc:1196] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/77ff391e67f14afd8609e842d33a1518
I20260810 03:19:44.053133 29426 ts_tablet_manager.cc:1939] T 77ff391e67f14afd8609e842d33a1518 P a34c00258bcd4542b0d3b32117a3c32f: Deleting consensus metadata
I20260810 03:19:44.055006 28753 catalog_manager.cc:4841] TS a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195:45863): tablet 77ff391e67f14afd8609e842d33a1518 (table test-table [id=e9faaa3f3305499892bd2c969f22b71a]) successfully deleted
I20260810 03:19:44.055624 29425 ts_tablet_manager.cc:1918] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:19:44.055816 29425 log.cc:1196] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/77ff391e67f14afd8609e842d33a1518
I20260810 03:19:44.055984 29240 tablet_service.cc:1467] Processing CreateTablet for tablet 66ad974296b04f66bc7d495ce773fd8d (DEFAULT_TABLE table=test-table [id=3c3904e07d7f4a46a1facf0d6403f8a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:44.056030 29085 tablet_service.cc:1467] Processing CreateTablet for tablet 66ad974296b04f66bc7d495ce773fd8d (DEFAULT_TABLE table=test-table [id=3c3904e07d7f4a46a1facf0d6403f8a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:44.056337 29425 ts_tablet_manager.cc:1939] T 77ff391e67f14afd8609e842d33a1518 P 9ded42fde8c240d29dd915484f5c3725: Deleting consensus metadata
I20260810 03:19:44.056497 29085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ad974296b04f66bc7d495ce773fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:44.056497 29240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ad974296b04f66bc7d495ce773fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:44.059672 28929 tablet_service.cc:1467] Processing CreateTablet for tablet 66ad974296b04f66bc7d495ce773fd8d (DEFAULT_TABLE table=test-table [id=3c3904e07d7f4a46a1facf0d6403f8a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:19:44.060248 28929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ad974296b04f66bc7d495ce773fd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:44.064049 28754 catalog_manager.cc:4841] TS 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259): tablet 77ff391e67f14afd8609e842d33a1518 (table test-table [id=e9faaa3f3305499892bd2c969f22b71a]) successfully deleted
I20260810 03:19:44.064237 29443 tablet_bootstrap.cc:492] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: Bootstrap starting.
I20260810 03:19:44.066754 29443 tablet_bootstrap.cc:654] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:44.069782 29442 tablet_bootstrap.cc:492] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: Bootstrap starting.
I20260810 03:19:44.070675 29441 tablet_bootstrap.cc:492] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: Bootstrap starting.
I20260810 03:19:44.072669 29441 tablet_bootstrap.cc:654] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:44.072705 29442 tablet_bootstrap.cc:654] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: Neither blocks nor log segments found. Creating new log.
I20260810 03:19:44.081187 29441 tablet_bootstrap.cc:492] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: No bootstrap required, opened a new log
I20260810 03:19:44.081414 29441 ts_tablet_manager.cc:1397] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260810 03:19:44.081879 29443 tablet_bootstrap.cc:492] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: No bootstrap required, opened a new log
I20260810 03:19:44.082088 29443 ts_tablet_manager.cc:1397] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260810 03:19:44.082244 29441 raft_consensus.cc:348] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.082516 29441 raft_consensus.cc:374] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:44.082638 29441 raft_consensus.cc:729] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475e306b3f074a04b92e3cbb90e94bae, State: Initialized, Role: FOLLOWER
I20260810 03:19:44.082911 29441 consensus_queue.cc:260] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.083472 29441 ts_tablet_manager.cc:1428] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 03:19:44.083333 29443 raft_consensus.cc:348] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.083653 29443 raft_consensus.cc:374] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:44.083743 29443 raft_consensus.cc:729] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Initialized, Role: FOLLOWER
I20260810 03:19:44.084000 29443 consensus_queue.cc:260] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.084604 29442 tablet_bootstrap.cc:492] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: No bootstrap required, opened a new log
I20260810 03:19:44.084754 29443 ts_tablet_manager.cc:1428] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 03:19:44.084805 29442 ts_tablet_manager.cc:1397] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20260810 03:19:44.085927 29442 raft_consensus.cc:348] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.086318 29442 raft_consensus.cc:374] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:44.086440 29442 raft_consensus.cc:729] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ded42fde8c240d29dd915484f5c3725, State: Initialized, Role: FOLLOWER
I20260810 03:19:44.086817 29442 consensus_queue.cc:260] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.087558 29442 ts_tablet_manager.cc:1428] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 03:19:44.222716 29362 raft_consensus.cc:482] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:19:44.222998 29362 raft_consensus.cc:504] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.223831 29362 leader_election.cc:288] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259), 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:44.224560 28955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ad974296b04f66bc7d495ce773fd8d" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ded42fde8c240d29dd915484f5c3725" is_pre_election: true
I20260810 03:19:44.224709 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ad974296b04f66bc7d495ce773fd8d" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae" is_pre_election: true
I20260810 03:19:44.224860 28955 raft_consensus.cc:2454] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 0.
I20260810 03:19:44.225155 29107 raft_consensus.cc:2454] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 0.
I20260810 03:19:44.225510 29184 leader_election.cc:302] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ded42fde8c240d29dd915484f5c3725, a34c00258bcd4542b0d3b32117a3c32f; no voters:
I20260810 03:19:44.225898 29362 raft_consensus.cc:2781] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:19:44.226017 29362 raft_consensus.cc:482] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:19:44.226178 29362 raft_consensus.cc:3037] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:44.228863 29362 raft_consensus.cc:504] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.229737 29362 leader_election.cc:288] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 election: Requested vote from peers 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259), 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893)
I20260810 03:19:44.230584 29107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ad974296b04f66bc7d495ce773fd8d" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475e306b3f074a04b92e3cbb90e94bae"
I20260810 03:19:44.230690 28955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ad974296b04f66bc7d495ce773fd8d" candidate_uuid: "a34c00258bcd4542b0d3b32117a3c32f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ded42fde8c240d29dd915484f5c3725"
I20260810 03:19:44.230981 29107 raft_consensus.cc:3037] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:44.230995 28955 raft_consensus.cc:3037] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:44.233393 28955 raft_consensus.cc:2454] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 1.
I20260810 03:19:44.233593 29107 raft_consensus.cc:2454] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c00258bcd4542b0d3b32117a3c32f in term 1.
I20260810 03:19:44.234055 29184 leader_election.cc:302] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ded42fde8c240d29dd915484f5c3725, a34c00258bcd4542b0d3b32117a3c32f; no voters:
I20260810 03:19:44.234476 29362 raft_consensus.cc:2781] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:44.234740 29362 raft_consensus.cc:686] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 1 LEADER]: Becoming Leader. State: Replica: a34c00258bcd4542b0d3b32117a3c32f, State: Running, Role: LEADER
I20260810 03:19:44.235064 29362 consensus_queue.cc:237] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } }
I20260810 03:19:44.238291 28770 catalog_manager.cc:5495] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f reported cstate change: term changed from 0 to 1, leader changed from <none> to a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195). New cstate: current_term: 1 leader_uuid: "a34c00258bcd4542b0d3b32117a3c32f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a34c00258bcd4542b0d3b32117a3c32f" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45863 } health_report { overall_health: HEALTHY } } }
I20260810 03:19:44.281215 29449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:44.274852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:47719 (local address 127.19.246.195:45863)
0810 03:19:44.275054 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:44.275061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:44.275676 (+ 615us) server_negotiation.cc:407] Connection header received
0810 03:19:44.275855 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:44.275861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:44.275946 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:44.276057 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:19:44.276514 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.277659 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.279149 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.279820 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.279952 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:44.280389 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:19:44.280538 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:19:44.280630 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:44.280770 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 03:19:44.280879 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":63}
I20260810 03:19:44.281220 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:44.274852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:45863 (local address 127.19.246.195:47719)
0810 03:19:44.275567 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:44.275584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:44.275723 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:44.276077 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:44.276080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:44.276089 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:19:44.276380 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:44.276388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.277821 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.277825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:44.278990 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:44.279004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.280033 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.280039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:44.280169 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:44.280215 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:19:44.280646 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:19:44.280655 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:19:44.280722 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 03:19:44.280863 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":1,"thread_start_us":387,"threads_started":1}
I20260810 03:19:44.292495 28955 raft_consensus.cc:1261] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Refusing update from remote peer a34c00258bcd4542b0d3b32117a3c32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:44.292511 29107 raft_consensus.cc:1261] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Refusing update from remote peer a34c00258bcd4542b0d3b32117a3c32f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:19:44.293255 29362 consensus_queue.cc:1035] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ded42fde8c240d29dd915484f5c3725" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:44.293591 29391 consensus_queue.cc:1035] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [LEADER]: Connected to new peer: Peer: permanent_uuid: "475e306b3f074a04b92e3cbb90e94bae" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:19:44.360411 28770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:44.360672 28770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:19:44.365406 28770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5634da3f9ec94ff5b3dba1cb50142952: Sending DeleteTablet for 3 replicas of tablet 66ad974296b04f66bc7d495ce773fd8d
I20260810 03:19:44.366554 29085 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ad974296b04f66bc7d495ce773fd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:44 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:50202
I20260810 03:19:44.366554 28929 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ad974296b04f66bc7d495ce773fd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:44 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60512
I20260810 03:19:44.366629 29239 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ad974296b04f66bc7d495ce773fd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:19:44 UTC) from {username='oryx', principal='oryx/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60140
I20260810 03:19:44.366925 29425 tablet_replica.cc:330] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: stopping tablet replica
I20260810 03:19:44.366984 29424 tablet_replica.cc:330] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: stopping tablet replica
I20260810 03:19:44.367134 29425 raft_consensus.cc:2229] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:44.367197 29426 tablet_replica.cc:330] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: stopping tablet replica
I20260810 03:19:44.367219 29424 raft_consensus.cc:2229] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:19:44.367336 29425 raft_consensus.cc:2258] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:44.367419 29426 raft_consensus.cc:2229] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:19:44.367497 29424 raft_consensus.cc:2258] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:44.367805 29426 raft_consensus.cc:2258] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:19:44.368762 29425 ts_tablet_manager.cc:1905] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:44.369186 29426 ts_tablet_manager.cc:1905] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:44.369642 29424 ts_tablet_manager.cc:1905] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:19:44.369864 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 28856
I20260810 03:19:44.377908 29426 ts_tablet_manager.cc:1918] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:19:44.378259 29426 log.cc:1196] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/66ad974296b04f66bc7d495ce773fd8d
I20260810 03:19:44.379174 29426 ts_tablet_manager.cc:1939] T 66ad974296b04f66bc7d495ce773fd8d P a34c00258bcd4542b0d3b32117a3c32f: Deleting consensus metadata
I20260810 03:19:44.380502 29424 ts_tablet_manager.cc:1918] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:19:44.380594 28753 catalog_manager.cc:4841] TS a34c00258bcd4542b0d3b32117a3c32f (127.19.246.195:45863): tablet 66ad974296b04f66bc7d495ce773fd8d (table test-table [id=3c3904e07d7f4a46a1facf0d6403f8a5]) successfully deleted
I20260810 03:19:44.380740 29424 log.cc:1196] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/66ad974296b04f66bc7d495ce773fd8d
I20260810 03:19:44.381292 29424 ts_tablet_manager.cc:1939] T 66ad974296b04f66bc7d495ce773fd8d P 475e306b3f074a04b92e3cbb90e94bae: Deleting consensus metadata
I20260810 03:19:44.381767 29425 ts_tablet_manager.cc:1918] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:19:44.381949 29425 log.cc:1196] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/66ad974296b04f66bc7d495ce773fd8d
I20260810 03:19:44.382618 29425 ts_tablet_manager.cc:1939] T 66ad974296b04f66bc7d495ce773fd8d P 9ded42fde8c240d29dd915484f5c3725: Deleting consensus metadata
I20260810 03:19:44.382714 28754 catalog_manager.cc:4841] TS 475e306b3f074a04b92e3cbb90e94bae (127.19.246.194:39893): tablet 66ad974296b04f66bc7d495ce773fd8d (table test-table [id=3c3904e07d7f4a46a1facf0d6403f8a5]) successfully deleted
I20260810 03:19:44.383821 28754 catalog_manager.cc:4841] TS 9ded42fde8c240d29dd915484f5c3725 (127.19.246.193:41259): tablet 66ad974296b04f66bc7d495ce773fd8d (table test-table [id=3c3904e07d7f4a46a1facf0d6403f8a5]) successfully deleted
Aug 10 03:19:44 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.193@KRBTEST.COM
I20260810 03:19:44.399645 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:44.371434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.19.246.193:41259)
0810 03:19:44.382379 (+ 10945us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:44.382388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:44.382412 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:19:44.382479 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:44.382485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:44.382598 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:44.382840 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:44.384749 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.385523 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.386875 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.387337 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.387536 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:44.392483 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:44.392503 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:44.392522 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:44.392571 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:44.396398 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:44.397299 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:44.397307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:44.397309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:44.397479 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:44.397904 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:44.397909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:44.397911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:44.398276 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:44.398632 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:44.398952 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 03:19:44.399211 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10790,"thread_start_us":10667,"threads_started":1}
I20260810 03:19:44.400391 29004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:44.536968 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 28856
I20260810 03:19:44.566504 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 29014
Aug 10 03:19:44 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.194@KRBTEST.COM
I20260810 03:19:44.585839 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:44.567472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.19.246.194:39893)
0810 03:19:44.568006 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:44.568011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:44.568027 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:19:44.568082 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:44.568085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:44.568150 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:44.568288 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:44.569778 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.570560 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.572548 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.573104 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.573233 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:44.578680 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:44.578699 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:44.578716 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:44.578780 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:44.582228 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:44.583269 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:44.583278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:44.583280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:44.583532 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:44.583983 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:44.583988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:44.583991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:44.584415 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:44.584749 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:44.585193 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 03:19:44.585458 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
I20260810 03:19:44.586597 29157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:44.690045 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 29014
I20260810 03:19:44.713995 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 29166
Aug 10 03:19:44 dist-test-slave-btw8 krb5kdc[28720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.246.195@KRBTEST.COM
I20260810 03:19:44.733518 29449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:44.714973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.19.246.195:45863)
0810 03:19:44.715161 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:44.715165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:44.715208 (+ 43us) server_negotiation.cc:407] Connection header received
0810 03:19:44.715266 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:44.715270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:44.715336 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:44.715482 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:44.717011 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.718024 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.719610 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:44.720275 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:44.720390 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:44.726033 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:44.726052 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:44.726070 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:44.726116 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:44.729716 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:44.730850 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:44.730858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:44.730860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:44.731037 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:44.731724 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:44.731728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:44.731730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:44.732179 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:44.732525 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:44.732882 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 03:19:44.733134 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:19:44.734275 29309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:44.896456 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 29166
I20260810 03:19:44.932570 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 28734
I20260810 03:19:44.934160 28820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:45.064692 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 28734
2026-08-10T03:19:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7861 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:19:45 dist-test-slave-btw8 krb5kdc[29481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:19:45 dist-test-slave-btw8 krb5kdc[29481](info): set up 2 sockets
Aug 10 03:19:45 dist-test-slave-btw8 krb5kdc[29481](info): commencing operation
krb5kdc: starting...
W20260810 03:19:46.326838 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.003s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:19:46 dist-test-slave-btw8 krb5kdc[29481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:19:46.673555 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:34149
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:34149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:47.105324 29496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:47.105691 29496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:47.105798 29496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:47.116057 29496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:47.116222 29496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:47.116276 29496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:47.116333 29496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:47.116380 29496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:47.128223 29496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:34149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:47.129700 29496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:47.131145 29496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:47.146903 29505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:47.146903 29503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:47.146886 29502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:47.148634 29496 server_base.cc:1034] running on GCE node
I20260810 03:19:47.149951 29496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:47.151978 29496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:47.153262 29496 hybrid_clock.cc:648] HybridClock initialized: now 1786331987153193 us; error 75 us; skew 500 ppm
I20260810 03:19:47.153826 29496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T03:19:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2161 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:19:47 dist-test-slave-btw8 krb5kdc[29511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:19:47 dist-test-slave-btw8 krb5kdc[29511](info): set up 2 sockets
Aug 10 03:19:47 dist-test-slave-btw8 krb5kdc[29511](info): commencing operation
krb5kdc: starting...
W20260810 03:19:48.582226 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.003s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:19:48 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:19:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:19:48.930135 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40097
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:40097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:49.328878 29525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:49.329236 29525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:49.329475 29525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:49.338075 29525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:49.338231 29525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:49.338284 29525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:49.338348 29525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:49.338461 29525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:49.349493 29525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:40097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:49.351008 29525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:49.352548 29525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:49.366611 29533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:49.366811 29530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:49.366784 29531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:49.368333 29525 server_base.cc:1034] running on GCE node
I20260810 03:19:49.369367 29525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:49.371088 29525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:49.372401 29525 hybrid_clock.cc:648] HybridClock initialized: now 1786331989372299 us; error 90 us; skew 500 ppm
I20260810 03:19:49.372850 29525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:49 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331989, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:49.377257 29525 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:19:49.379568 29525 webserver.cc:460] Webserver started at http://127.19.246.254:43495/ using document root <none> and password file <none>
I20260810 03:19:49.380420 29525 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:49.380550 29525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:49.380859 29525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:49.385684 29525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "63e674c5821042c1b68b94eeb31db8f1"
format_stamp: "Formatted at 2026-08-10 03:19:49 on dist-test-slave-btw8"
server_key: "dfef39d5b03f705c7baf28659a13e5cd"
server_key_iv: "e5bab38e7cc19222d4496a98eb981f63"
server_key_version: "encryptionkey@0"
I20260810 03:19:49.386619 29525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "63e674c5821042c1b68b94eeb31db8f1"
format_stamp: "Formatted at 2026-08-10 03:19:49 on dist-test-slave-btw8"
server_key: "dfef39d5b03f705c7baf28659a13e5cd"
server_key_iv: "e5bab38e7cc19222d4496a98eb981f63"
server_key_version: "encryptionkey@0"
I20260810 03:19:49.393911 29525 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260810 03:19:49.399253 29542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:49.401041 29525 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:49.401248 29525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "63e674c5821042c1b68b94eeb31db8f1"
format_stamp: "Formatted at 2026-08-10 03:19:49 on dist-test-slave-btw8"
server_key: "dfef39d5b03f705c7baf28659a13e5cd"
server_key_iv: "e5bab38e7cc19222d4496a98eb981f63"
server_key_version: "encryptionkey@0"
I20260810 03:19:49.401464 29525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:49.429876 29525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:49.435451 29525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:49.435761 29525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:49.458654 29525 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:40097
I20260810 03:19:49.458676 29601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:40097 every 8 connection(s)
I20260810 03:19:49.460400 29525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:49.465011 29602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:19:49.467057 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 29525
I20260810 03:19:49.467518 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:19:49.468448 20443 external_mini_cluster.cc:1428] Setting key f5c513ff9a155a765185024fb039cfe7
I20260810 03:19:49.481046 29602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Bootstrap starting.
I20260810 03:19:49.486485 29602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Neither blocks nor log segments found. Creating new log.
Aug 10 03:19:49 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:49.487907 29602 log.cc:824] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:49.492190 29602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: No bootstrap required, opened a new log
I20260810 03:19:49.499224 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:49.471079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.19.246.254:40097)
0810 03:19:49.472137 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:49.472151 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:49.472216 (+ 65us) server_negotiation.cc:407] Connection header received
0810 03:19:49.474182 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:49.474287 (+ 105us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:49.474777 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:49.475170 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:49.477600 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:49.479317 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:49.481604 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:49.482415 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:49.482593 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:49.488930 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:49.488942 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:49.488964 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:49.489110 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:49.493304 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:49.494496 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:49.494505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:49.494513 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:49.494750 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:49.495229 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:49.495236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:49.495244 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:49.495994 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:49.496494 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:49.497308 (+ 814us) server_negotiation.cc:299] Negotiation successful
0810 03:19:49.497857 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":512,"threads_started":1}
I20260810 03:19:49.502233 29602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } }
I20260810 03:19:49.502889 29602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:19:49.503036 29602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e674c5821042c1b68b94eeb31db8f1, State: Initialized, Role: FOLLOWER
I20260810 03:19:49.503829 29602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } }
I20260810 03:19:49.504092 29602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:49.504204 29602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:49.504345 29602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:19:49.507061 29602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } }
I20260810 03:19:49.507818 29602 leader_election.cc:302] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e674c5821042c1b68b94eeb31db8f1; no voters:
I20260810 03:19:49.508639 29602 leader_election.cc:288] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:19:49.508823 29608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:19:49.509956 29608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 1 LEADER]: Becoming Leader. State: Replica: 63e674c5821042c1b68b94eeb31db8f1, State: Running, Role: LEADER
I20260810 03:19:49.510783 29608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } }
I20260810 03:19:49.511524 29602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:49.514022 29610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e674c5821042c1b68b94eeb31db8f1. Latest consensus state: current_term: 1 leader_uuid: "63e674c5821042c1b68b94eeb31db8f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } } }
I20260810 03:19:49.514457 29610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:49.515578 29613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:49.515408 29609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e674c5821042c1b68b94eeb31db8f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } } }
I20260810 03:19:49.516656 29609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:49.523486 29613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:49.534301 29613 catalog_manager.cc:1347] Generated new cluster ID: c103e798460e430aa5719bcf44d06320
I20260810 03:19:49.534469 29613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:49.574769 29613 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:19:49.576480 29613 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:49.584957 29613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Generated new TSK 0
I20260810 03:19:49.586037 29613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:19:49.676517 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40097
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:50.084758 29631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:50.085166 29631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:50.085346 29631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:50.094116 29631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:50.094439 29631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:50.094817 29631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:50.107209 29631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:50.108719 29631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:50.110157 29631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:50.126767 29637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:50.127265 29636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:50.127310 29640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:50.127831 29631 server_base.cc:1034] running on GCE node
I20260810 03:19:50.128610 29631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:50.130059 29631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:50.131412 29631 hybrid_clock.cc:648] HybridClock initialized: now 1786331990131291 us; error 59 us; skew 500 ppm
I20260810 03:19:50.131809 29631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:50 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331990, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:50.135998 29631 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:19:50.137921 29631 webserver.cc:460] Webserver started at http://127.19.246.193:40899/ using document root <none> and password file <none>
I20260810 03:19:50.138790 29631 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:50.138978 29631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:50.139417 29631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:50.142971 29631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "8586e92bd495449ea19e6a2f706b650d"
format_stamp: "Formatted at 2026-08-10 03:19:50 on dist-test-slave-btw8"
server_key: "ccc84e4e1dc27bb7db82e70fbdb6f52c"
server_key_iv: "c5d4d1f85863a37bbb84407e68bd41fd"
server_key_version: "encryptionkey@0"
I20260810 03:19:50.144032 29631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "8586e92bd495449ea19e6a2f706b650d"
format_stamp: "Formatted at 2026-08-10 03:19:50 on dist-test-slave-btw8"
server_key: "ccc84e4e1dc27bb7db82e70fbdb6f52c"
server_key_iv: "c5d4d1f85863a37bbb84407e68bd41fd"
server_key_version: "encryptionkey@0"
I20260810 03:19:50.150626 29631 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:19:50.155205 29647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:50.157160 29631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:19:50.157425 29631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "8586e92bd495449ea19e6a2f706b650d"
format_stamp: "Formatted at 2026-08-10 03:19:50 on dist-test-slave-btw8"
server_key: "ccc84e4e1dc27bb7db82e70fbdb6f52c"
server_key_iv: "c5d4d1f85863a37bbb84407e68bd41fd"
server_key_version: "encryptionkey@0"
I20260810 03:19:50.157686 29631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:50.171977 29631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:50.177279 29631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:50.177645 29631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:50.178771 29631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:50.181143 29631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:50.181622 29631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260810 03:19:50.181856 29631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:50.181941 29631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:50.223507 29631 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:40163
I20260810 03:19:50.223531 29775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:40163 every 8 connection(s)
I20260810 03:19:50.225040 29631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:50.231686 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 29631
I20260810 03:19:50.231935 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:19:50.232810 20443 external_mini_cluster.cc:1428] Setting key e6e2646437e8519df1a8cd25979cdf06
Aug 10 03:19:50 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331990, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:50.253193 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:50.228311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:46943 (local address 127.19.246.254:40097)
0810 03:19:50.228984 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:50.228989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:50.230270 (+ 1281us) server_negotiation.cc:407] Connection header received
0810 03:19:50.233314 (+ 3044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:50.233321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:50.233428 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:50.233624 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:50.236028 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:50.236764 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:50.238559 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:50.239000 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:50.239162 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:50.243942 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:50.243949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:50.243953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:50.243995 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:50.247646 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:50.248882 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:50.248891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:50.248898 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:50.249097 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:50.249899 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:50.249908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:50.249914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:50.250387 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:50.250589 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:50.252276 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0810 03:19:50.252652 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":404,"threads_started":1}
I20260810 03:19:50.254333 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:50.229153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:46943)
0810 03:19:50.230020 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:50.230113 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:50.232753 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:50.233991 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:50.234053 (+ 62us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:50.234954 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:50.235809 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:50.235831 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:50.236948 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:50.236952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:50.238323 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:50.238357 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:50.239304 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:50.239311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:50.239642 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:50.240472 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:50.240498 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:50.243558 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:50.247908 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:50.247917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:50.247936 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:50.248688 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:50.249430 (+ 742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:50.249434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:50.249437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:50.249681 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:50.250692 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:50.250702 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:50.251592 (+ 890us) client_negotiation.cc:769] Sending connection context
0810 03:19:50.252017 (+ 425us) client_negotiation.cc:240] Negotiation successful
0810 03:19:50.252850 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":365,"threads_started":1}
I20260810 03:19:50.256708 29776 heartbeater.cc:347] Connected to a master server at 127.19.246.254:40097
I20260810 03:19:50.257125 29776 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:50.258082 29776 heartbeater.cc:511] Master 127.19.246.254:40097 requested a full tablet report, sending...
I20260810 03:19:50.260919 29563 ts_manager.cc:194] Registered new tserver with Master: 8586e92bd495449ea19e6a2f706b650d (127.19.246.193:40163)
I20260810 03:19:50.263016 29563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:46943
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:19:50.303592 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40097
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:50.688840 29783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:50.689245 29783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:50.689415 29783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:50.698526 29783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:50.698746 29783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:50.698967 29783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:50.710682 29783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:50.712590 29783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:50.714874 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:50.732362 29788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:50.732250 29789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:50.732281 29791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:50.733492 29783 server_base.cc:1034] running on GCE node
I20260810 03:19:50.734587 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:50.736014 29783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:50.737317 29783 hybrid_clock.cc:648] HybridClock initialized: now 1786331990737275 us; error 42 us; skew 500 ppm
I20260810 03:19:50.737795 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:50 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331990, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:50.742044 29783 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:19:50.744020 29783 webserver.cc:460] Webserver started at http://127.19.246.194:34695/ using document root <none> and password file <none>
I20260810 03:19:50.744866 29783 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:50.745041 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:50.745863 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:50.750085 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "8c8a4ed26fc74c8aaba4298b9e88154d"
format_stamp: "Formatted at 2026-08-10 03:19:50 on dist-test-slave-btw8"
server_key: "11cf661678be3857cea70c4bdce53184"
server_key_iv: "c77c5499b5b14a3f511873abbfeea86f"
server_key_version: "encryptionkey@0"
I20260810 03:19:50.751199 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "8c8a4ed26fc74c8aaba4298b9e88154d"
format_stamp: "Formatted at 2026-08-10 03:19:50 on dist-test-slave-btw8"
server_key: "11cf661678be3857cea70c4bdce53184"
server_key_iv: "c77c5499b5b14a3f511873abbfeea86f"
server_key_version: "encryptionkey@0"
I20260810 03:19:50.757939 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:19:50.762542 29799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:50.764928 29783 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:19:50.765188 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "8c8a4ed26fc74c8aaba4298b9e88154d"
format_stamp: "Formatted at 2026-08-10 03:19:50 on dist-test-slave-btw8"
server_key: "11cf661678be3857cea70c4bdce53184"
server_key_iv: "c77c5499b5b14a3f511873abbfeea86f"
server_key_version: "encryptionkey@0"
I20260810 03:19:50.765525 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:50.797497 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:50.802529 29783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:50.802879 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:50.804059 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:50.805725 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:50.805807 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:50.805984 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:50.806061 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:50.842808 29783 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:43077
I20260810 03:19:50.842871 29925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:43077 every 8 connection(s)
I20260810 03:19:50.844323 29783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:50.849578 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 29783
I20260810 03:19:50.849835 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:19:50.850462 20443 external_mini_cluster.cc:1428] Setting key 3be54c3c5294127de48d2661f6cf1bae
Aug 10 03:19:50 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331990, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:50.872651 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:50.847606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:40719 (local address 127.19.246.254:40097)
0810 03:19:50.848259 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:50.848264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:50.849313 (+ 1049us) server_negotiation.cc:407] Connection header received
0810 03:19:50.852308 (+ 2995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:50.852314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:50.852398 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:50.852619 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:50.855753 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:50.856519 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:50.859743 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:50.860364 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:50.860506 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:50.865809 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:50.865815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:50.865819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:50.865860 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:50.868831 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:50.869898 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:50.869903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:50.869906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:50.870042 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:50.870568 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:50.870573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:50.870576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:50.870926 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:50.871097 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:50.871990 (+ 893us) server_negotiation.cc:299] Negotiation successful
0810 03:19:50.872262 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":387,"threads_started":1}
I20260810 03:19:50.873759 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:50.848105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:40719)
0810 03:19:50.849067 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:50.849136 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:50.851979 (+ 2843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:50.853008 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:50.853044 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:50.854013 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:50.855402 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:50.855444 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:50.856740 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:50.856749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:50.859508 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:50.859535 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:50.860677 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:50.860688 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:50.861084 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:50.862232 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:50.862272 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:50.865390 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:50.869060 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:50.869072 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:50.869100 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:50.869737 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:50.870174 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:50.870179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:50.870183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:50.870399 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:50.871067 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:50.871076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:50.871567 (+ 491us) client_negotiation.cc:769] Sending connection context
0810 03:19:50.871867 (+ 300us) client_negotiation.cc:240] Negotiation successful
0810 03:19:50.872496 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":352,"threads_started":1}
I20260810 03:19:50.875458 29926 heartbeater.cc:347] Connected to a master server at 127.19.246.254:40097
I20260810 03:19:50.875861 29926 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:50.876891 29926 heartbeater.cc:511] Master 127.19.246.254:40097 requested a full tablet report, sending...
I20260810 03:19:50.878785 29563 ts_manager.cc:194] Registered new tserver with Master: 8c8a4ed26fc74c8aaba4298b9e88154d (127.19.246.194:43077)
I20260810 03:19:50.879761 29563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:40719
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:19:50.927266 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40097
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:51.267800 29776 heartbeater.cc:503] Master 127.19.246.254:40097 was elected leader, sending a full tablet report...
W20260810 03:19:51.333891 29932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:51.334388 29932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:51.334631 29932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:51.344923 29932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:51.345067 29932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:51.345247 29932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:51.358655 29932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:40097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:51.360292 29932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:51.361801 29932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:51.380657 29937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:51.380921 29932 server_base.cc:1034] running on GCE node
W20260810 03:19:51.380679 29941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:51.380465 29938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:51.382577 29932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:51.384023 29932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:51.385340 29932 hybrid_clock.cc:648] HybridClock initialized: now 1786331991385265 us; error 79 us; skew 500 ppm
I20260810 03:19:51.385775 29932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:51 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331991, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:51.389796 29932 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:19:51.391867 29932 webserver.cc:460] Webserver started at http://127.19.246.195:33715/ using document root <none> and password file <none>
I20260810 03:19:51.392657 29932 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:51.392795 29932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:51.393101 29932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:19:51.397068 29932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "b77360d60df147ffa8c2a03c1b437bdd"
format_stamp: "Formatted at 2026-08-10 03:19:51 on dist-test-slave-btw8"
server_key: "3d770357cded0789c833832244b43300"
server_key_iv: "618c1a4a71794e520f2843aa5352228c"
server_key_version: "encryptionkey@0"
I20260810 03:19:51.398097 29932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "b77360d60df147ffa8c2a03c1b437bdd"
format_stamp: "Formatted at 2026-08-10 03:19:51 on dist-test-slave-btw8"
server_key: "3d770357cded0789c833832244b43300"
server_key_iv: "618c1a4a71794e520f2843aa5352228c"
server_key_version: "encryptionkey@0"
I20260810 03:19:51.405740 29932 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260810 03:19:51.411720 29948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:51.413944 29932 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260810 03:19:51.414314 29932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "b77360d60df147ffa8c2a03c1b437bdd"
format_stamp: "Formatted at 2026-08-10 03:19:51 on dist-test-slave-btw8"
server_key: "3d770357cded0789c833832244b43300"
server_key_iv: "618c1a4a71794e520f2843aa5352228c"
server_key_version: "encryptionkey@0"
I20260810 03:19:51.414600 29932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:51.438926 29932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:51.446161 29932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:51.447044 29932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:51.448577 29932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:51.450390 29932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:51.450556 29932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:51.450752 29932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:51.450840 29932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:51.498647 29932 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:35943
I20260810 03:19:51.498847 30067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:35943 every 8 connection(s)
I20260810 03:19:51.500170 29932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:51.509107 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 29932
I20260810 03:19:51.509353 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:19:51.509927 20443 external_mini_cluster.cc:1428] Setting key 175d297de7c72da3e219a9086e9e192a
Aug 10 03:19:51 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331991, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:19:51.528483 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:51.503531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:58131 (local address 127.19.246.254:40097)
0810 03:19:51.504107 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:51.504112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:51.505426 (+ 1314us) server_negotiation.cc:407] Connection header received
0810 03:19:51.507681 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:51.507687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:51.507821 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:51.507952 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:51.510543 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:51.511975 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:51.514476 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:51.515028 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:51.515170 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:51.519952 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:51.519958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:51.519962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:51.520005 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:51.524161 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:51.525383 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:51.525414 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:51.525417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:51.525647 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:51.526281 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:51.526289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:51.526294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:51.526649 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:51.526855 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:51.527815 (+ 960us) server_negotiation.cc:299] Negotiation successful
0810 03:19:51.528085 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":331,"threads_started":1}
I20260810 03:19:51.529589 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:51.504293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:58131)
0810 03:19:51.505217 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:51.505263 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:51.507292 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:51.508239 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:51.508249 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:51.509074 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:51.510006 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:51.510051 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:51.512253 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:51.512258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:51.514259 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:51.514279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:51.515339 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:51.515345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:51.515678 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:51.516575 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:51.516608 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:51.519528 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:19:51.524401 (+ 4873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:51.524413 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:51.524435 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:51.525245 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:51.525802 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:19:51.525807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:19:51.525810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:19:51.526109 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:19:51.526800 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:19:51.526810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:19:51.527318 (+ 508us) client_negotiation.cc:769] Sending connection context
0810 03:19:51.527700 (+ 382us) client_negotiation.cc:240] Negotiation successful
0810 03:19:51.528316 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":373,"threads_started":1}
I20260810 03:19:51.531833 30068 heartbeater.cc:347] Connected to a master server at 127.19.246.254:40097
I20260810 03:19:51.532168 30068 heartbeater.cc:464] Registering TS with master...
I20260810 03:19:51.532965 30068 heartbeater.cc:511] Master 127.19.246.254:40097 requested a full tablet report, sending...
I20260810 03:19:51.534829 29563 ts_manager.cc:194] Registered new tserver with Master: b77360d60df147ffa8c2a03c1b437bdd (127.19.246.195:35943)
I20260810 03:19:51.535895 29563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:58131
I20260810 03:19:51.540707 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab.
Entry for principal oryx/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab.
WARNING: no policy specified for oryx/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab.
Entry for principal oryx/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab.
WARNING: no policy specified for oryx/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab.
Entry for principal oryx/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab.
I20260810 03:19:51.756795 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 29631
Aug 10 03:19:51 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:19:51.779171 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:51.758242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.19.246.193:40163)
0810 03:19:51.759189 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:51.759197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:51.759257 (+ 60us) server_negotiation.cc:407] Connection header received
0810 03:19:51.759326 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:51.759334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:51.759682 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:51.759832 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:51.761488 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:51.762833 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:51.764677 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:51.765388 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:51.765517 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:51.771076 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:51.771102 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:51.771121 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:51.771190 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:51.775194 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:51.776285 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:51.776295 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:51.776297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:51.776505 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:51.777175 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:51.777179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:51.777181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:51.777622 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:51.777912 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:51.778410 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 03:19:51.778686 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":464,"threads_started":1}
I20260810 03:19:51.780912 29770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:51.872808 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 29631
I20260810 03:19:51.883945 29926 heartbeater.cc:503] Master 127.19.246.254:40097 was elected leader, sending a full tablet report...
I20260810 03:19:51.895746 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 29783
Aug 10 03:19:51 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:19:51.921808 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:51.897036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.19.246.194:43077)
0810 03:19:51.898020 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:51.898028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:51.898056 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:19:51.898123 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:51.898131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:51.898431 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:51.898771 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:51.901278 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:51.903316 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:51.906154 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:51.907207 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:51.907489 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:51.913845 (+ 6356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:51.913869 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:51.913888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:51.913945 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:51.917571 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:51.918810 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:51.918819 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:51.918822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:51.919053 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:51.919630 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:51.919635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:51.919638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:51.920199 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:51.920602 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:51.920895 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 03:19:51.921425 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":674,"threads_started":1}
I20260810 03:19:51.923507 29920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:52.023926 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 29783
I20260810 03:19:52.046895 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 29932
Aug 10 03:19:52 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:19:52.069471 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:52.048124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.19.246.195:35943)
0810 03:19:52.048868 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:52.048876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:52.048907 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:19:52.048980 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:52.048987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:52.049235 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:52.049489 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:52.051035 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:52.052239 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:52.053736 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:52.054482 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:52.054680 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:52.061545 (+ 6865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:52.061567 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:52.061584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:52.061634 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:52.065291 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:52.066381 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:52.066391 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:52.066393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:52.066583 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:52.067046 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:52.067052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:52.067056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:52.067921 (+ 865us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:52.068368 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:52.068681 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 03:19:52.069033 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":335,"threads_started":1}
I20260810 03:19:52.071077 30063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:52.168879 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 29932
I20260810 03:19:52.192076 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 29525
I20260810 03:19:52.194235 29597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:19:52.311244 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 29525
I20260810 03:19:52.336280 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40097
--webserver_interface=127.19.246.254
--webserver_port=43495
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:40097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:19:52.744932 30086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:52.745276 30086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:52.745458 30086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:52.753803 30086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:19:52.753973 30086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:52.754067 30086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:52.754175 30086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:19:52.754276 30086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:19:52.766811 30086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:40097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=43495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:52.768446 30086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:52.769937 30086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:52.784792 30095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:52.784866 30092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:52.784874 30093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:52.785710 30086 server_base.cc:1034] running on GCE node
I20260810 03:19:52.786633 30086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:52.788434 30086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:52.789779 30086 hybrid_clock.cc:648] HybridClock initialized: now 1786331992789693 us; error 66 us; skew 500 ppm
I20260810 03:19:52.790346 30086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:52 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331992, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:52.802377 30086 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:19:52.804442 30086 webserver.cc:460] Webserver started at http://127.19.246.254:43495/ using document root <none> and password file <none>
I20260810 03:19:52.805327 30086 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:52.805477 30086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:52.812572 30086 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260810 03:19:52.816841 30103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:52.818622 30086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:19:52.818940 30086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "63e674c5821042c1b68b94eeb31db8f1"
format_stamp: "Formatted at 2026-08-10 03:19:49 on dist-test-slave-btw8"
server_key: "dfef39d5b03f705c7baf28659a13e5cd"
server_key_iv: "e5bab38e7cc19222d4496a98eb981f63"
server_key_version: "encryptionkey@0"
I20260810 03:19:52.821171 30086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:52.838201 30086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:52.845446 30086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:52.845746 30086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:52.870023 30086 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:40097
I20260810 03:19:52.870041 30161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:40097 every 8 connection(s)
I20260810 03:19:52.872058 30086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:19:52.873668 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 30086
I20260810 03:19:52.874601 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:40163
--local_ip_for_outbound_sockets=127.19.246.193
--tserver_master_addrs=127.19.246.254:40097
--webserver_port=40899
--webserver_interface=127.19.246.193
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.246.193
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:19:52.881573 30162 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:19:52.886778 30162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Bootstrap starting.
I20260810 03:19:52.912766 30162 log.cc:824] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:19:52.924331 30162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:19:52.925006 30162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Bootstrap complete.
I20260810 03:19:52.935735 30162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } }
I20260810 03:19:52.936730 30162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e674c5821042c1b68b94eeb31db8f1, State: Initialized, Role: FOLLOWER
I20260810 03:19:52.937917 30162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } }
I20260810 03:19:52.938287 30162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:19:52.938469 30162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:19:52.938745 30162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:19:52.944612 30162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } }
I20260810 03:19:52.945340 30162 leader_election.cc:302] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e674c5821042c1b68b94eeb31db8f1; no voters:
I20260810 03:19:52.946108 30162 leader_election.cc:288] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:19:52.946283 30167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:19:52.947945 30167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [term 2 LEADER]: Becoming Leader. State: Replica: 63e674c5821042c1b68b94eeb31db8f1, State: Running, Role: LEADER
I20260810 03:19:52.948710 30167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } }
I20260810 03:19:52.949054 30162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:19:52.952189 30168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "63e674c5821042c1b68b94eeb31db8f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } } }
I20260810 03:19:52.952620 30168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:52.953294 30169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e674c5821042c1b68b94eeb31db8f1. Latest consensus state: current_term: 2 leader_uuid: "63e674c5821042c1b68b94eeb31db8f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e674c5821042c1b68b94eeb31db8f1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40097 } } }
I20260810 03:19:52.953709 30169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:19:52.965358 30173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:19:52.981230 30173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:19:52.985626 30173 catalog_manager.cc:1259] Loaded cluster ID: c103e798460e430aa5719bcf44d06320
I20260810 03:19:52.986274 30173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:19:52.992905 30173 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:19:52.994998 30173 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 63e674c5821042c1b68b94eeb31db8f1: Loaded TSK: 0
I20260810 03:19:52.997052 30173 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:19:53.334743 30164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:53.335091 30164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:53.335274 30164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:53.345136 30164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:53.345304 30164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:53.345507 30164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:19:53.359025 30164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.193.keytab
--principal=oryx/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:40163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=40899
--tserver_master_addrs=127.19.246.254:40097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:53.360917 30164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:53.363341 30164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:53.384397 30193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:53.384383 30194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:53.389339 30196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:53.391425 30164 server_base.cc:1034] running on GCE node
I20260810 03:19:53.392342 30164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:53.393882 30164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:53.395447 30164 hybrid_clock.cc:648] HybridClock initialized: now 1786331993395268 us; error 176 us; skew 500 ppm
I20260810 03:19:53.395879 30164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:53 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331993, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:53.400624 30164 init.cc:377] Logged in from keytab as oryx/127.19.246.193@KRBTEST.COM (short username oryx)
I20260810 03:19:53.403321 30164 webserver.cc:460] Webserver started at http://127.19.246.193:40899/ using document root <none> and password file <none>
I20260810 03:19:53.404618 30164 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:53.404866 30164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:53.412695 30164 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260810 03:19:53.419615 30203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:53.423300 30164 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260810 03:19:53.423666 30164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "8586e92bd495449ea19e6a2f706b650d"
format_stamp: "Formatted at 2026-08-10 03:19:50 on dist-test-slave-btw8"
server_key: "ccc84e4e1dc27bb7db82e70fbdb6f52c"
server_key_iv: "c5d4d1f85863a37bbb84407e68bd41fd"
server_key_version: "encryptionkey@0"
I20260810 03:19:53.425271 30164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:53.449720 30164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:53.458700 30164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:53.459087 30164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:53.461351 30164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:53.464423 30164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:53.464570 30164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:53.464893 30164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:53.465246 30164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:53.514096 30164 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:40163
I20260810 03:19:53.514180 30325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:40163 every 8 connection(s)
I20260810 03:19:53.516203 30164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:19:53.525393 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 30164
I20260810 03:19:53.526237 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:43077
--local_ip_for_outbound_sockets=127.19.246.194
--tserver_master_addrs=127.19.246.254:40097
--webserver_port=34695
--webserver_interface=127.19.246.194
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.246.194
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:19:53 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:53 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:53.543107 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:53.520284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:38749)
0810 03:19:53.521336 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:53.521389 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:53.523324 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:53.525229 (+ 1905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:53.525243 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:53.526050 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:53.526863 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:53.526903 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.529331 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.529337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:53.531075 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:53.531100 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.533057 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.533070 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:53.533503 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:53.534552 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:53.534596 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:53.541184 (+ 6588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":436,"threads_started":1}
W20260810 03:19:53.543614 30328 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:53.519981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:38749 (local address 127.19.246.254:40097)
0810 03:19:53.520901 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:53.520920 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:53.521575 (+ 655us) server_negotiation.cc:407] Connection header received
0810 03:19:53.524123 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:53.524138 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:53.524561 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:53.525021 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:53.527143 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.528885 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.531343 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.532711 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.532948 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:53.541981 (+ 9033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:38749: BlockingRecv error: recv got EOF from 127.19.246.193:38749 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":363,"threads_started":1}
W20260810 03:19:53.545111 30326 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:40097 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.246.254:40097: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Aug 10 03:19:53 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:53 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:53.559700 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:53.546328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:56815)
0810 03:19:53.546628 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:53.546660 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:53.546806 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:53.547254 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:53.547262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:53.547868 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:53.548151 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:53.548172 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.550092 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.550103 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:53.552356 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:53.552394 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.553392 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.553402 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:53.553671 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:53.554847 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:53.554882 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:53.559315 (+ 4433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 03:19:53.560425 30328 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:53.546678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:56815 (local address 127.19.246.254:40097)
0810 03:19:53.546902 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:53.546907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:53.546931 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:19:53.547002 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:53.547006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:53.547078 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:53.547188 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:53.548526 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.549827 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.552675 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.553162 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.553299 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:53.559657 (+ 6358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:56815: BlockingRecv error: recv got EOF from 127.19.246.193:56815 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 03:19:53 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:53 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:53.575052 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:53.560977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:50133)
0810 03:19:53.561223 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:53.561252 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:53.561405 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:53.562163 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:53.562172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:53.562868 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:53.563307 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:53.563332 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.565404 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.565409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:53.566728 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:53.566751 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.567955 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.567960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:53.568403 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:53.569317 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:53.569340 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:53.574779 (+ 5439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 03:19:53.575496 30328 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:53.561070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:50133 (local address 127.19.246.254:40097)
0810 03:19:53.561393 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:53.561403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:53.561438 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:19:53.561545 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:53.561554 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:53.561700 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:53.561845 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:53.563660 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.565067 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.566964 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:53.567737 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:53.567900 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:53.575162 (+ 7262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:50133: BlockingRecv error: recv got EOF from 127.19.246.193:50133 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":2663,"threads_started":1}
W20260810 03:19:53.575676 30326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:19:53.959338 30331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:53.959815 30331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:53.959988 30331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:53.969522 30331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:53.969703 30331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:53.969905 30331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:19:53.982801 30331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.194.keytab
--principal=oryx/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:43077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=34695
--tserver_master_addrs=127.19.246.254:40097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:53.984485 30331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:53.985951 30331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:19:54.004204 30339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:54.004689 30338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:54.006237 30341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:54.007339 30331 server_base.cc:1034] running on GCE node
I20260810 03:19:54.009059 30331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:54.010845 30331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:54.012511 30331 hybrid_clock.cc:648] HybridClock initialized: now 1786331994012427 us; error 77 us; skew 500 ppm
I20260810 03:19:54.013130 30331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331994, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:54.018797 30331 init.cc:377] Logged in from keytab as oryx/127.19.246.194@KRBTEST.COM (short username oryx)
I20260810 03:19:54.021447 30331 webserver.cc:460] Webserver started at http://127.19.246.194:34695/ using document root <none> and password file <none>
I20260810 03:19:54.022595 30331 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:54.022827 30331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:54.032802 30331 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 03:19:54.036384 30348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:54.038317 30331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:19:54.038599 30331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "8c8a4ed26fc74c8aaba4298b9e88154d"
format_stamp: "Formatted at 2026-08-10 03:19:50 on dist-test-slave-btw8"
server_key: "11cf661678be3857cea70c4bdce53184"
server_key_iv: "c77c5499b5b14a3f511873abbfeea86f"
server_key_version: "encryptionkey@0"
I20260810 03:19:54.040118 30331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:54.063735 30331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:54.070629 30331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:54.071038 30331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:54.072809 30331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:54.075320 30331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:54.075498 30331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 03:19:54.076056 30331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:54.076172 30331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:54.122195 30331 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:43077
I20260810 03:19:54.122274 30473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:43077 every 8 connection(s)
I20260810 03:19:54.124014 30331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:19:54.132584 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 30331
I20260810 03:19:54.133525 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:35943
--local_ip_for_outbound_sockets=127.19.246.195
--tserver_master_addrs=127.19.246.254:40097
--webserver_port=33715
--webserver_interface=127.19.246.195
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.246.195
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:19:54.152086 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:54.127686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:43975 (local address 127.19.246.254:40097)
0810 03:19:54.128245 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:54.128250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:54.129534 (+ 1284us) server_negotiation.cc:407] Connection header received
0810 03:19:54.131910 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:54.131916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:54.132066 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:54.132465 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:54.135635 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.136650 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.138780 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.139273 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.139474 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.151809 (+ 12335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:43975: BlockingRecv error: recv got EOF from 127.19.246.194:43975 (error 108)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
I20260810 03:19:54.153038 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:54.128466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:43975)
0810 03:19:54.129322 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:54.129377 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:54.131531 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:54.133049 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:54.133064 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:54.134095 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:54.135043 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.135085 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.136939 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.136945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.138541 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.138562 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.139598 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.139611 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.139918 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.143896 (+ 3978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:54.143977 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:54.151343 (+ 7366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":345,"threads_started":1}
W20260810 03:19:54.154219 30474 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:40097 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.246.254:40097: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:54.169631 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:54.155290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:52177)
0810 03:19:54.155626 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:54.155651 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:54.155842 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:54.156669 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:54.156674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:54.157385 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:54.157810 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.157827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.159508 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.159513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.161088 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.161134 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.163103 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.163112 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.163687 (+ 575us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.164882 (+ 1195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:54.164913 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:54.169272 (+ 4359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260810 03:19:54.169925 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:54.155449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:52177 (local address 127.19.246.254:40097)
0810 03:19:54.155649 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:54.155656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:54.155749 (+ 93us) server_negotiation.cc:407] Connection header received
0810 03:19:54.155982 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:54.155989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:54.156317 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:54.156603 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:54.158263 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.159225 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.161666 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.162747 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.163039 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.169617 (+ 6578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:52177: BlockingRecv error: recv got EOF from 127.19.246.194:52177 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:54.180917 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:54.170910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:38923)
0810 03:19:54.171157 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:54.171181 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:54.171481 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:54.171940 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:54.171945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:54.172413 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:54.172612 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.172623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.173826 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.173835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.175056 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.175073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.176040 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.176047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.176271 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.176988 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:54.177016 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:54.180581 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 03:19:54.181373 30474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:19:54.181941 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:54.171035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:38923 (local address 127.19.246.254:40097)
0810 03:19:54.171444 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:54.171459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:54.171490 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:19:54.171689 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:54.171693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:54.171765 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:54.171905 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:54.172825 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.173646 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.175268 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.175851 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.176013 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.181061 (+ 5048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:38923: BlockingRecv error: recv got EOF from 127.19.246.194:38923 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 03:19:54.551055 30478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:19:54.551738 30478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:19:54.551934 30478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:19:54.561290 30478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:19:54.561520 30478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:19:54.561729 30478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:19:54.572665 30478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/krb5kdc/oryx_127.19.246.195.keytab
--principal=oryx/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:35943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=33715
--tserver_master_addrs=127.19.246.254:40097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:19:54.574208 30478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:19:54.575749 30478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:54.586987 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:54.576480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:38341)
0810 03:19:54.577164 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:54.577182 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:54.577369 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:54.577742 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:54.577747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:54.578369 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:54.578579 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.578591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.579781 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.579793 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.580822 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.580835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.581653 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.581659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.581841 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.582580 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:54.582604 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:54.586633 (+ 4029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":405,"threads_started":1}
W20260810 03:19:54.587167 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:54.576862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:38341 (local address 127.19.246.254:40097)
0810 03:19:54.577077 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:54.577083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:54.577288 (+ 205us) server_negotiation.cc:407] Connection header received
0810 03:19:54.577516 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:54.577520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:54.577593 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:54.577705 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:54.578753 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.579610 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.580990 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.581443 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.581612 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.586911 (+ 5299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:38341: BlockingRecv error: recv got EOF from 127.19.246.193:38341 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260810 03:19:54.592262 30488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:54.592348 30487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:19:54.592388 30491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:19:54.593418 30478 server_base.cc:1034] running on GCE node
I20260810 03:19:54.594166 30478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:19:54.595577 30478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:19:54.596844 30478 hybrid_clock.cc:648] HybridClock initialized: now 1786331994596783 us; error 69 us; skew 500 ppm
I20260810 03:19:54.597237 30478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786331994, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:19:54.601949 30478 init.cc:377] Logged in from keytab as oryx/127.19.246.195@KRBTEST.COM (short username oryx)
I20260810 03:19:54.604012 30478 webserver.cc:460] Webserver started at http://127.19.246.195:33715/ using document root <none> and password file <none>
I20260810 03:19:54.604957 30478 fs_manager.cc:359] Metadata directory not provided
I20260810 03:19:54.605185 30478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:19:54.612715 30478 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 03:19:54.616466 30498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:54.618697 30478 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:19:54.618980 30478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "b77360d60df147ffa8c2a03c1b437bdd"
format_stamp: "Formatted at 2026-08-10 03:19:51 on dist-test-slave-btw8"
server_key: "3d770357cded0789c833832244b43300"
server_key_iv: "618c1a4a71794e520f2843aa5352228c"
server_key_version: "encryptionkey@0"
I20260810 03:19:54.620338 30478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:19:54.644712 30478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:19:54.650250 30478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:19:54.650601 30478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:19:54.651774 30478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:19:54.653535 30478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:19:54.653640 30478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:54.653725 30478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:19:54.653784 30478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:19:54.695930 30478 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:35943
I20260810 03:19:54.696012 30620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:35943 every 8 connection(s)
I20260810 03:19:54.697616 30478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:19:54.708020 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 30478
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:19:54.719826 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:54.701585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:33947 (local address 127.19.246.254:40097)
0810 03:19:54.701884 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:54.701892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:54.703076 (+ 1184us) server_negotiation.cc:407] Connection header received
0810 03:19:54.705622 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:54.705652 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:54.705785 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:54.705917 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:54.708282 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.709529 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.711585 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.712360 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.712930 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.719522 (+ 6592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:33947: BlockingRecv error: recv got EOF from 127.19.246.195:33947 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
I20260810 03:19:54.720734 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:54.701842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:33947)
0810 03:19:54.702845 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:54.702906 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:54.704893 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:54.706333 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:54.706345 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:54.707057 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:54.707902 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.707924 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.709788 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.709798 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.711339 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.711419 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.712908 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.712914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.713176 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.714035 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:54.714082 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:54.719173 (+ 5091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":427,"threads_started":1}
W20260810 03:19:54.721650 30622 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:40097 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.246.254:40097: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
I20260810 03:19:54.730644 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:54.709148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.19.246.254:40097)
0810 03:19:54.709731 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:54.709735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:54.710298 (+ 563us) server_negotiation.cc:407] Connection header received
0810 03:19:54.710520 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:54.710525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:54.710597 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:54.710717 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:54.713276 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.714322 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.716010 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.716488 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.716643 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.720721 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:19:54.720732 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:19:54.720752 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:19:54.720844 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:19:54.725775 (+ 4931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:54.726995 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:54.727004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:54.727013 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:54.727411 (+ 398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:19:54.727929 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:19:54.727935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:19:54.727937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:19:54.728682 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:19:54.729041 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0810 03:19:54.729728 (+ 687us) server_negotiation.cc:299] Negotiation successful
0810 03:19:54.730213 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":262,"threads_started":1}
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:54.733494 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:54.722343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:48297)
0810 03:19:54.722598 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:54.722623 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:54.722759 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:54.723164 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:54.723171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:54.723741 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:54.724002 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.724020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.725493 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.725502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.726792 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.726808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.727981 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.727988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.728240 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.729004 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:54.729029 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:54.733180 (+ 4151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260810 03:19:54.733808 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:54.722522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:48297 (local address 127.19.246.254:40097)
0810 03:19:54.722820 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:54.722825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:54.722846 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:19:54.722906 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:54.722910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:54.722981 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:54.723095 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:54.724289 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.725317 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.727012 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.727795 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.727924 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.733400 (+ 5476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:48297: BlockingRecv error: recv got EOF from 127.19.246.195:48297 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:54 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:54.745779 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:54.734434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:46085)
0810 03:19:54.734615 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:54.734647 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:54.734765 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:54.735100 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:54.735106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:54.735585 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:54.735760 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.735770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.737021 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.737025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.738082 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:54.738096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.738965 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.738971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:54.739210 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.740251 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:54.740285 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:54.745375 (+ 5090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 03:19:54.746011 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:54.734531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:46085 (local address 127.19.246.254:40097)
0810 03:19:54.734732 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:54.734741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:54.734763 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:19:54.734873 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:54.734877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:54.734940 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:54.735068 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:54.735922 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.736763 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.738274 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:54.738775 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:54.738931 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:54.745741 (+ 6810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:46085: BlockingRecv error: recv got EOF from 127.19.246.195:46085 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 03:19:54.746457 30622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 03:19:55 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:55 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:55.194927 30628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:55.182167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:42081)
0810 03:19:55.182942 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:55.182966 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:55.183113 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:55.184196 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:55.184202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:55.184941 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:55.185257 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:55.185273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.187189 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.187197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:55.188284 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:55.188301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.189424 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.189430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:55.189646 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:55.190519 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:55.190539 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:55.194652 (+ 4113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":447,"threads_started":1}
W20260810 03:19:55.195664 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:55.182430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:42081 (local address 127.19.246.254:40097)
0810 03:19:55.182709 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:55.182716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:55.183073 (+ 357us) server_negotiation.cc:407] Connection header received
0810 03:19:55.183594 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:55.183602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:55.183716 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:55.184121 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:55.185634 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.186941 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.188475 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.189013 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.189132 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:55.195123 (+ 5991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:42081: BlockingRecv error: recv got EOF from 127.19.246.194:42081 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 10 03:19:55 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:55 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:55.601065 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:55.588300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:35445)
0810 03:19:55.589002 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:55.589022 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:55.589237 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:55.589731 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:55.589737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:55.590312 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:55.590674 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:55.590690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.592332 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.592338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:55.593786 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:55.593812 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.594845 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.594853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:55.595046 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:55.595958 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:55.595979 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:55.600778 (+ 4799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":416,"threads_started":1}
W20260810 03:19:55.601468 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:55.588527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:35445 (local address 127.19.246.254:40097)
0810 03:19:55.588824 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:55.588830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:55.589257 (+ 427us) server_negotiation.cc:407] Connection header received
0810 03:19:55.589346 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:55.589351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:55.589453 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:55.589634 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:55.590968 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.592038 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.594108 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.594599 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.594829 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:55.601101 (+ 6272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:35445: BlockingRecv error: recv got EOF from 127.19.246.193:35445 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Aug 10 03:19:55 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:55 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:55.757573 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:55.747109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:58429)
0810 03:19:55.747727 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:55.747744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:55.747925 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:55.748290 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:55.748296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:55.748923 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:55.749231 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:55.749244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.750721 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.750725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:55.751897 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:55.751915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.752839 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.752847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:55.753026 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:55.753743 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:55.753765 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:55.757275 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":347,"threads_started":1}
W20260810 03:19:55.757956 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:55.747197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:58429 (local address 127.19.246.254:40097)
0810 03:19:55.747526 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:55.747535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:55.747837 (+ 302us) server_negotiation.cc:407] Connection header received
0810 03:19:55.748057 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:55.748062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:55.748137 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:55.748254 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:55.749394 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.750513 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.752072 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:55.752617 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:55.752764 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:55.757665 (+ 4901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:58429: BlockingRecv error: recv got EOF from 127.19.246.195:58429 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 10 03:19:56 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:56 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:56.208325 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:56.196208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:50425)
0810 03:19:56.197079 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:56.197112 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:56.197329 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:56.198121 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:56.198126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:56.198682 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:56.198924 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:56.198937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.200446 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.200455 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:56.201648 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:56.201668 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.202681 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.202687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:56.202837 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:56.203662 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:56.203687 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:56.207904 (+ 4217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":462,"threads_started":1}
W20260810 03:19:56.208680 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:56.196212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:50425 (local address 127.19.246.254:40097)
0810 03:19:56.196446 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:56.196458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:56.197303 (+ 845us) server_negotiation.cc:407] Connection header received
0810 03:19:56.197741 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:56.197747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:56.197831 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:56.197950 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:56.199184 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.200176 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.201860 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.202444 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.202624 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:56.208246 (+ 5622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:50425: BlockingRecv error: recv got EOF from 127.19.246.194:50425 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 03:19:56 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:56 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:56.612793 30633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:56.602339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:58827)
0810 03:19:56.602947 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:56.602967 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:56.603108 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:56.603930 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:56.603934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:56.604356 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:56.604553 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:56.604570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.605811 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.605815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:56.606903 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:56.606918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.607764 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.607771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:56.607928 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:56.608667 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:56.608688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:56.612486 (+ 3798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":352,"threads_started":1}
W20260810 03:19:56.613212 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:56.602633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:58827 (local address 127.19.246.254:40097)
0810 03:19:56.602885 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:56.602890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:56.603049 (+ 159us) server_negotiation.cc:407] Connection header received
0810 03:19:56.603466 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:56.603472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:56.603584 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:56.603866 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:56.604726 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.605563 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.607054 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.607608 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.607798 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:56.612932 (+ 5134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:58827: BlockingRecv error: recv got EOF from 127.19.246.193:58827 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 03:19:56 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:56 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:56.769639 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:56.758818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:60245)
0810 03:19:56.759468 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:56.759486 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:56.759621 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:56.760003 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:56.760006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:56.760493 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:56.760701 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:56.760711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.761906 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.761910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:56.763159 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:56.763178 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.764175 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.764181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:56.764318 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:56.765056 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:56.765077 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:56.769363 (+ 4286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":360,"threads_started":1}
W20260810 03:19:56.769883 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:56.759022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:60245 (local address 127.19.246.254:40097)
0810 03:19:56.759227 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:56.759232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:56.759584 (+ 352us) server_negotiation.cc:407] Connection header received
0810 03:19:56.759768 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:56.759773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:56.759839 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:56.759935 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:56.760878 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.761726 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.763417 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:56.763952 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:56.764081 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:56.769675 (+ 5594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:60245: BlockingRecv error: recv got EOF from 127.19.246.195:60245 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 03:19:57 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:57 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:57.221160 30635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:57.210055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:39943)
0810 03:19:57.210722 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:57.210747 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:57.210969 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:57.211455 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:57.211461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:57.211911 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:57.212114 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:57.212124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.213341 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.213348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:57.214457 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:57.214473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.215437 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.215445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:57.215677 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:57.216414 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:57.216433 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:57.220671 (+ 4238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":335,"threads_started":1}
W20260810 03:19:57.221166 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:57.210082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:39943 (local address 127.19.246.254:40097)
0810 03:19:57.210350 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:57.210361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:57.210885 (+ 524us) server_negotiation.cc:407] Connection header received
0810 03:19:57.211129 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:57.211136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:57.211241 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:57.211393 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:57.212305 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.213160 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.214653 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.215191 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.215404 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:57.220925 (+ 5521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:39943: BlockingRecv error: recv got EOF from 127.19.246.194:39943 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 10 03:19:57 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:57 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:57.626066 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:57.613867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:58029)
0810 03:19:57.614651 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:57.614683 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:57.615074 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:57.615770 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:57.615775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:57.616320 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:57.616502 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:57.616513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.617839 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.617844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:57.619531 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:57.619553 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.620443 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.620449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:57.620602 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:57.621503 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:57.621526 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:57.625615 (+ 4089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":392,"threads_started":1}
W20260810 03:19:57.626479 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:57.614291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:58029 (local address 127.19.246.254:40097)
0810 03:19:57.614653 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:57.614664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:57.614951 (+ 287us) server_negotiation.cc:407] Connection header received
0810 03:19:57.615287 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:57.615295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:57.615503 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:57.615644 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:57.616763 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.617630 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.619767 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.620255 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.620371 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:57.626007 (+ 5636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:58029: BlockingRecv error: recv got EOF from 127.19.246.193:58029 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Aug 10 03:19:57 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:57 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:57.781867 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:57.771166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:34123)
0810 03:19:57.772066 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:57.772088 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:57.772309 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:57.772825 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:57.772832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:57.773354 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:57.773566 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:57.773576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.774913 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.774921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:57.776067 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:57.776085 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.776998 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.777003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:57.777144 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:57.777901 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:57.777919 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:57.781561 (+ 3642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":507,"threads_started":1}
W20260810 03:19:57.782287 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:57.771448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:34123 (local address 127.19.246.254:40097)
0810 03:19:57.771787 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:57.771797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:57.772217 (+ 420us) server_negotiation.cc:407] Connection header received
0810 03:19:57.772463 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:57.772471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:57.772550 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:57.772722 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:57.773789 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.774729 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.776286 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:57.776820 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:57.776949 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:57.781932 (+ 4983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:34123: BlockingRecv error: recv got EOF from 127.19.246.195:34123 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 10 03:19:58 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:58 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:58.239127 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:58.222514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:50535)
0810 03:19:58.223140 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:58.223162 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:58.223297 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:58.223759 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:58.223763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:58.224423 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:58.224726 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:58.224738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.226267 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.226273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:58.227799 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:58.227821 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.228801 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.228806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:58.228949 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:58.234628 (+ 5679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:58.234653 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:58.238856 (+ 4203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":367,"threads_started":1}
W20260810 03:19:58.239656 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:58.222691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:50535 (local address 127.19.246.254:40097)
0810 03:19:58.222922 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:58.222927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:58.223285 (+ 358us) server_negotiation.cc:407] Connection header received
0810 03:19:58.223491 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:58.223495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:58.223567 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:58.223738 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:58.224884 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.226074 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.228020 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.228601 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.228750 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:58.239255 (+ 10505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:50535: BlockingRecv error: recv got EOF from 127.19.246.194:50535 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 10 03:19:58 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:58 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:58.638163 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:58.627627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:38679)
0810 03:19:58.628230 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:58.628247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:58.628404 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:58.628858 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:58.628861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:58.629397 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:58.629576 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:58.629585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.631028 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.631033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:58.632340 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:58.632362 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.633522 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.633526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:58.633671 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:58.634417 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:58.634436 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:58.637834 (+ 3398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":302,"threads_started":1}
W20260810 03:19:58.638644 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:58.627696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:38679 (local address 127.19.246.254:40097)
0810 03:19:58.627879 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:58.627885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:58.628320 (+ 435us) server_negotiation.cc:407] Connection header received
0810 03:19:58.628540 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:58.628548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:58.628637 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:58.628745 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:58.629913 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.630856 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.632566 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.633346 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.633467 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:58.638186 (+ 4719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:38679: BlockingRecv error: recv got EOF from 127.19.246.193:38679 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 03:19:58 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:58 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:58.794227 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:58.783188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:35165)
0810 03:19:58.783903 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:58.783935 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:58.784106 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:58.784535 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:58.784550 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:58.784976 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:58.785146 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:58.785156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.786569 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.786574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:58.788291 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:58.788312 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.789221 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.789230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:58.789440 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:58.790176 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:58.790209 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:58.793844 (+ 3635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":471,"threads_started":1}
W20260810 03:19:58.794531 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:58.783279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:35165 (local address 127.19.246.254:40097)
0810 03:19:58.783626 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:58.783631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:58.784109 (+ 478us) server_negotiation.cc:407] Connection header received
0810 03:19:58.784232 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:58.784237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:58.784311 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:58.784421 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:58.785358 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.786357 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.788472 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:58.789012 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:58.789186 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:58.794180 (+ 4994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:35165: BlockingRecv error: recv got EOF from 127.19.246.195:35165 (error 108)
Metrics: {"server-negotiator.queue_time_us":205}
Aug 10 03:19:59 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:59 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:59.251281 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:59.240545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:47805)
0810 03:19:59.241407 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:59.241438 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:59.241566 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:59.242003 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:59.242008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:59.242544 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:59.242723 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:59.242733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.243953 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.243959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:59.244916 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:59.244930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.245807 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.245813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:59.245993 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:59.246769 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:59.246789 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:59.250974 (+ 4185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":320,"threads_started":1}
W20260810 03:19:59.251617 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:59.240885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:47805 (local address 127.19.246.254:40097)
0810 03:19:59.241117 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:59.241126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:59.241528 (+ 402us) server_negotiation.cc:407] Connection header received
0810 03:19:59.241740 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:59.241746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:59.241837 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:59.241966 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:59.242932 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.243795 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.245087 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.245638 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.245776 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:59.251294 (+ 5518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:47805: BlockingRecv error: recv got EOF from 127.19.246.194:47805 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 03:19:59 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:59 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:59.649940 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:59.639475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:47693)
0810 03:19:59.640125 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:59.640143 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:59.640279 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:59.640769 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:59.640773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:59.641254 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:59.641447 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:59.641456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.642684 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.642688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:59.643833 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:59.643850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.644946 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.644956 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:59.645181 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:59.646120 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:59.646151 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:59.649704 (+ 3553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":390,"threads_started":1}
W20260810 03:19:59.650281 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:59.639827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:47693 (local address 127.19.246.254:40097)
0810 03:19:59.640064 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:59.640071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:59.640253 (+ 182us) server_negotiation.cc:407] Connection header received
0810 03:19:59.640389 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:59.640394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:59.640462 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:59.640554 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:59.641626 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.642500 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.644035 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.644637 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.644787 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:59.649970 (+ 5183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:47693: BlockingRecv error: recv got EOF from 127.19.246.193:47693 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 10 03:19:59 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:19:59 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:19:59.806605 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:19:59.795795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:59895)
0810 03:19:59.796365 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 03:19:59.796384 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:19:59.796566 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:19:59.796942 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:19:59.796948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:19:59.797519 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:19:59.797801 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:59.797815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.799002 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.799008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:59.800162 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:19:59.800177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.801065 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.801072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:19:59.801271 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:59.802590 (+ 1319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:19:59.802614 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:19:59.806199 (+ 3585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":276,"threads_started":1}
W20260810 03:19:59.806775 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:19:59.795866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:59895 (local address 127.19.246.254:40097)
0810 03:19:59.796091 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 03:19:59.796132 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 03:19:59.796485 (+ 353us) server_negotiation.cc:407] Connection header received
0810 03:19:59.796710 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:19:59.796716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:19:59.796796 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:19:59.796907 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:19:59.797978 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.798830 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.800343 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:19:59.800883 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:19:59.801044 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:19:59.806509 (+ 5465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:59895: BlockingRecv error: recv got EOF from 127.19.246.195:59895 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 03:20:00 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:00 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:00.264163 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:00.252704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:57479)
0810 03:20:00.253544 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:00.253567 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:00.253735 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:00.254118 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:00.254121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:00.254513 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:00.254670 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:00.254678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.256556 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.256562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:00.257825 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:00.257846 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.258861 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.258867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:00.259013 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:00.259967 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:00.259988 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:00.263826 (+ 3838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":587,"threads_started":1}
W20260810 03:20:00.264581 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:00.252845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:57479 (local address 127.19.246.254:40097)
0810 03:20:00.253107 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:00.253116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:00.253657 (+ 541us) server_negotiation.cc:407] Connection header received
0810 03:20:00.253892 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:00.253898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:00.253981 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:00.254072 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:00.255000 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.256190 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.258089 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.258615 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.258738 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:00.264258 (+ 5520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:57479: BlockingRecv error: recv got EOF from 127.19.246.194:57479 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 03:20:00 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:00 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:00.661623 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:00.651427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:38001)
0810 03:20:00.652149 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:00.652168 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:00.652304 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:00.652716 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:00.652721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:00.653249 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:00.653442 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:00.653453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.654622 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.654634 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:00.655737 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:00.655753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.656619 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.656626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:00.656828 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:00.657628 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:00.657647 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:00.661305 (+ 3658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":308,"threads_started":1}
W20260810 03:20:00.661929 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:00.651479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:38001 (local address 127.19.246.254:40097)
0810 03:20:00.651703 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:00.651710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:00.652277 (+ 567us) server_negotiation.cc:407] Connection header received
0810 03:20:00.652461 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:00.652467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:00.652552 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:00.652677 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:00.653617 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.654421 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.655934 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.656409 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.656589 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:00.661626 (+ 5037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:38001: BlockingRecv error: recv got EOF from 127.19.246.193:38001 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 03:20:00 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:00 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:00.817811 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:00.807945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:40915)
0810 03:20:00.808625 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:00.808644 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:00.808843 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:00.809220 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:00.809224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:00.809665 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:00.809825 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:00.809834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.810957 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.810962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:00.812070 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:00.812086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.812953 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.812958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:00.813090 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:00.813953 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:00.813975 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:00.817537 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":425,"threads_started":1}
W20260810 03:20:00.818351 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:00.808134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:40915 (local address 127.19.246.254:40097)
0810 03:20:00.808417 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:00.808426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:00.808762 (+ 336us) server_negotiation.cc:407] Connection header received
0810 03:20:00.808991 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:00.808995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:00.809070 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:00.809183 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:00.809982 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.810785 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.812274 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:00.812778 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:00.812904 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:00.817826 (+ 4922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:40915: BlockingRecv error: recv got EOF from 127.19.246.195:40915 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 10 03:20:01 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:01 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:01.279130 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:01.265949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:52111)
0810 03:20:01.267748 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:01.267801 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:01.268000 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:01.268516 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:01.268520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:01.269028 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:01.269240 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:01.269251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.270607 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.270613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:01.272239 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:01.272263 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.273355 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.273361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:01.273513 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:01.274836 (+ 1323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:01.274860 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:01.278813 (+ 3953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":748,"threads_started":1}
W20260810 03:20:01.279831 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:01.266030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:52111 (local address 127.19.246.254:40097)
0810 03:20:01.266690 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:01.266702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:01.267961 (+ 1259us) server_negotiation.cc:407] Connection header received
0810 03:20:01.268143 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:01.268147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:01.268282 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:01.268386 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:01.269490 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.270347 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.272504 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.273090 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.273245 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:01.279471 (+ 6226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:52111: BlockingRecv error: recv got EOF from 127.19.246.194:52111 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Aug 10 03:20:01 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:01 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:01.678278 30651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:01.663228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:32861)
0810 03:20:01.664306 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:01.664520 (+ 214us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:01.664702 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:01.665629 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:01.665634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:01.666375 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:01.666636 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:01.666651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.668821 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.668828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:01.670039 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:01.670062 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.671996 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.672018 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:01.672298 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:01.673636 (+ 1338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:01.673667 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:01.677966 (+ 4299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":737,"threads_started":1}
W20260810 03:20:01.679026 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:01.663416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:32861 (local address 127.19.246.254:40097)
0810 03:20:01.663734 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:01.663747 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:01.664698 (+ 951us) server_negotiation.cc:407] Connection header received
0810 03:20:01.664977 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:01.664982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:01.665309 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:01.665435 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:01.666946 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.668490 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.670402 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.671338 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.671544 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:01.678678 (+ 7134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:32861: BlockingRecv error: recv got EOF from 127.19.246.193:32861 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 10 03:20:01 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:01 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:01.832971 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:01.819117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:36433)
0810 03:20:01.819978 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:01.819998 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:01.820147 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:01.820526 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:01.820530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:01.821039 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:01.821279 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:01.821290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.823015 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.823021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:01.824337 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:01.824357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.825799 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.825809 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:01.826061 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:01.827408 (+ 1347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:01.827443 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:01.832669 (+ 5226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":481,"threads_started":1}
W20260810 03:20:01.833534 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:01.819504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:36433 (local address 127.19.246.254:40097)
0810 03:20:01.819726 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:01.819731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:01.820126 (+ 395us) server_negotiation.cc:407] Connection header received
0810 03:20:01.820274 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:01.820279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:01.820355 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:01.820458 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:01.821578 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.822667 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.824729 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:01.825396 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:01.825601 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:01.833249 (+ 7648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:36433: BlockingRecv error: recv got EOF from 127.19.246.195:36433 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 03:20:02 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:02 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:02.292917 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:02.280778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:34031)
0810 03:20:02.281398 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:02.281423 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:02.281629 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:02.282120 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:02.282126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:02.282743 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:02.282935 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:02.282946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.284811 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.284818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:02.285928 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:02.285947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.287140 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.287147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:02.287422 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:02.288613 (+ 1191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:02.288646 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:02.292657 (+ 4011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":423,"threads_started":1}
W20260810 03:20:02.293376 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:02.281053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:34031 (local address 127.19.246.254:40097)
0810 03:20:02.281339 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:02.281347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:02.281542 (+ 195us) server_negotiation.cc:407] Connection header received
0810 03:20:02.281851 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:02.281857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:02.281931 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:02.282040 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:02.283225 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.284601 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.286293 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.286877 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.287059 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:02.293023 (+ 5964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:34031: BlockingRecv error: recv got EOF from 127.19.246.194:34031 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 10 03:20:02 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:02 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:02.692476 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:02.679737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:32827)
0810 03:20:02.680509 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:02.680529 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:02.680753 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:02.681288 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:02.681299 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:02.681982 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:02.682252 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:02.682265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.683751 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.683758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:02.685605 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:02.685626 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.686577 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.686583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:02.686817 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:02.687681 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:02.687709 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:02.692041 (+ 4332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":407,"threads_started":1}
W20260810 03:20:02.692596 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:02.679875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:32827 (local address 127.19.246.254:40097)
0810 03:20:02.680220 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:02.680230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:02.680663 (+ 433us) server_negotiation.cc:407] Connection header received
0810 03:20:02.680933 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:02.680937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:02.681017 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:02.681144 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:02.682446 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.683500 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.685827 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.686355 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.686472 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:02.692315 (+ 5843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:32827: BlockingRecv error: recv got EOF from 127.19.246.193:32827 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Aug 10 03:20:02 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:02 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:02.849131 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:02.834282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:37303)
0810 03:20:02.835118 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:02.835152 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:02.835336 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:02.836383 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:02.836389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:02.836983 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:02.837251 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:02.837267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.839469 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.839476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:02.840847 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:02.840874 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.841909 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.841915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:02.842058 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:02.842885 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:02.842907 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:02.848711 (+ 5804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":480,"threads_started":1}
W20260810 03:20:02.849678 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:02.834517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:37303 (local address 127.19.246.254:40097)
0810 03:20:02.834811 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:02.834820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:02.835470 (+ 650us) server_negotiation.cc:407] Connection header received
0810 03:20:02.835633 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:02.835640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:02.836115 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:02.836296 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:02.837528 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.839072 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.841083 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:02.841607 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:02.841732 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:02.849232 (+ 7500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:37303: BlockingRecv error: recv got EOF from 127.19.246.195:37303 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 10 03:20:03 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:03 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:03.310618 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:03.294294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:50183)
0810 03:20:03.295430 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:03.295481 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:03.295714 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:03.296135 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:03.296142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:03.296712 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:03.296909 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:03.296920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.298126 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.298132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:03.299462 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:03.299479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.300600 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.300607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:03.300818 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:03.305990 (+ 5172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:03.306029 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:03.310215 (+ 4186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":969,"thread_start_us":808,"threads_started":1}
W20260810 03:20:03.310781 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:03.294409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:50183 (local address 127.19.246.254:40097)
0810 03:20:03.295110 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:03.295122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:03.295628 (+ 506us) server_negotiation.cc:407] Connection header received
0810 03:20:03.295869 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:03.295880 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:03.295960 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:03.296085 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:03.297125 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.297929 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.299636 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.300404 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.300567 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:03.310494 (+ 9927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:50183: BlockingRecv error: recv got EOF from 127.19.246.194:50183 (error 108)
Metrics: {"server-negotiator.queue_time_us":239}
Aug 10 03:20:03 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:03 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:03.704262 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:03.693839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:54211)
0810 03:20:03.694495 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:03.694516 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:03.694675 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:03.695075 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:03.695080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:03.695687 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:03.695846 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:03.695855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.697255 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.697259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:03.698265 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:03.698279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.699478 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.699485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:03.699639 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:03.700435 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:03.700455 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:03.703977 (+ 3522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":395,"threads_started":1}
W20260810 03:20:03.704668 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:03.693921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:54211 (local address 127.19.246.254:40097)
0810 03:20:03.694165 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:03.694172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:03.694690 (+ 518us) server_negotiation.cc:407] Connection header received
0810 03:20:03.694800 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:03.694808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:03.694901 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:03.694990 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:03.695989 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.697082 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.698434 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.699141 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.699282 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:03.704292 (+ 5010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:54211: BlockingRecv error: recv got EOF from 127.19.246.193:54211 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 03:20:03 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:03 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:03.861303 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:03.850474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:53063)
0810 03:20:03.851158 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:03.851179 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:03.851342 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:03.851849 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:03.851853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:03.852322 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:03.852526 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:03.852535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.854213 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.854221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:03.855472 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:03.855491 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.856438 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.856446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:03.856664 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:03.857459 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:03.857480 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:03.861020 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":365,"threads_started":1}
W20260810 03:20:03.861693 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:03.850686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:53063 (local address 127.19.246.254:40097)
0810 03:20:03.850936 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:03.850941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:03.851321 (+ 380us) server_negotiation.cc:407] Connection header received
0810 03:20:03.851550 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:03.851555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:03.851639 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:03.851760 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:03.852760 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.854033 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.855674 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:03.856230 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:03.856402 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:03.861424 (+ 5022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:53063: BlockingRecv error: recv got EOF from 127.19.246.195:53063 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 03:20:04 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:04 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:04.325265 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:04.312216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:39371)
0810 03:20:04.313099 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:04.313131 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:04.313291 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:04.313809 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:04.313813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:04.314305 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:04.314609 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:04.314625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.316646 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.316657 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:04.318687 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:04.318724 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.319821 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.319830 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:04.320081 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:04.321076 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:04.321100 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:04.324978 (+ 3878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":597,"threads_started":1}
W20260810 03:20:04.325798 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:04.312403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:39371 (local address 127.19.246.254:40097)
0810 03:20:04.312607 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:04.312612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:04.313423 (+ 811us) server_negotiation.cc:407] Connection header received
0810 03:20:04.313501 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:04.313505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:04.313580 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:04.313701 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:04.314951 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.316267 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.318923 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.319502 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.319637 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:04.325375 (+ 5738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:39371: BlockingRecv error: recv got EOF from 127.19.246.194:39371 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 03:20:04 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:04 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:04.719313 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:04.705456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:40441)
0810 03:20:04.706194 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:04.706214 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:04.706360 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:04.707146 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:04.707152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:04.708089 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:04.708388 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:04.708405 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.710267 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.710275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:04.711679 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:04.711708 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.713023 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.713031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:04.713256 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:04.714145 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:04.714167 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:04.719015 (+ 4848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":455,"threads_started":1}
W20260810 03:20:04.719795 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:04.705717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:40441 (local address 127.19.246.254:40097)
0810 03:20:04.705969 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:04.705974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:04.706497 (+ 523us) server_negotiation.cc:407] Connection header received
0810 03:20:04.706634 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:04.706639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:04.706854 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:04.706974 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:04.708808 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.709927 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.712008 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.712666 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.712832 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:04.719424 (+ 6592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:40441: BlockingRecv error: recv got EOF from 127.19.246.193:40441 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 10 03:20:04 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:04 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:04.873137 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:04.862765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:52219)
0810 03:20:04.863497 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:04.863538 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:04.863726 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:04.864160 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:04.864165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:04.864762 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:04.864951 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:04.864961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.866184 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.866191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:04.867272 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:04.867286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.868291 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.868300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:04.868504 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:04.869243 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:04.869265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:04.872753 (+ 3488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":422,"threads_started":1}
W20260810 03:20:04.873411 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:04.862871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:52219 (local address 127.19.246.254:40097)
0810 03:20:04.863201 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:04.863220 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:04.863623 (+ 403us) server_negotiation.cc:407] Connection header received
0810 03:20:04.863908 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:04.863913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:04.863989 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:04.864118 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:04.865138 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.866005 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.867527 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:04.868085 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:04.868252 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:04.873035 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:52219: BlockingRecv error: recv got EOF from 127.19.246.195:52219 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 10 03:20:05 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:05 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:05.340394 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:05.327210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:58201)
0810 03:20:05.328091 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:05.328122 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:05.328304 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:05.329053 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:05.329059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:05.329730 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:05.330036 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:05.330051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.332032 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.332045 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:05.333444 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:05.333470 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.334497 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.334504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:05.334666 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:05.335634 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:05.335656 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:05.340031 (+ 4375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":423,"threads_started":1}
W20260810 03:20:05.340750 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:05.327267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:58201 (local address 127.19.246.254:40097)
0810 03:20:05.327531 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:05.327541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:05.328295 (+ 754us) server_negotiation.cc:407] Connection header received
0810 03:20:05.328564 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:05.328569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:05.328647 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:05.328838 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:05.330407 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.331644 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.333700 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.334267 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.334401 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:05.340363 (+ 5962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:58201: BlockingRecv error: recv got EOF from 127.19.246.194:58201 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 10 03:20:05 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:05 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:05.731985 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:05.720641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:54787)
0810 03:20:05.721352 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:05.721379 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:05.721643 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:05.722084 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:05.722090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:05.722639 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:05.722827 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:05.722836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.724230 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.724241 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:05.725662 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:05.725682 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.726721 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.726728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:05.726947 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:05.727842 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:05.727863 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:05.731681 (+ 3818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":443,"threads_started":1}
W20260810 03:20:05.732484 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:05.720940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:54787 (local address 127.19.246.254:40097)
0810 03:20:05.721228 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:05.721235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:05.721483 (+ 248us) server_negotiation.cc:407] Connection header received
0810 03:20:05.721807 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:05.721815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:05.721894 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:05.722089 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:05.723040 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.724028 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.725837 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.726520 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.726685 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:05.731999 (+ 5314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:54787: BlockingRecv error: recv got EOF from 127.19.246.193:54787 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 10 03:20:05 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:05 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:05.885871 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:05.874503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:36699)
0810 03:20:05.875088 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:05.875110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:05.875278 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:05.875737 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:05.875743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:05.876325 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:05.876575 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:05.876591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.877983 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.877990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:05.879609 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:05.879632 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.880694 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.880702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:05.880904 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:05.881768 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:05.881795 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:05.885495 (+ 3700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":299,"threads_started":1}
W20260810 03:20:05.886050 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:05.874633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:36699 (local address 127.19.246.254:40097)
0810 03:20:05.874886 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:05.874893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:05.875239 (+ 346us) server_negotiation.cc:407] Connection header received
0810 03:20:05.875465 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:05.875471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:05.875558 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:05.875647 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:05.876758 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.877801 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.879873 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:05.880466 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:05.880611 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:05.885784 (+ 5173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:36699: BlockingRecv error: recv got EOF from 127.19.246.195:36699 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 03:20:06 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:06 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:06.357313 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:06.342443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:43847)
0810 03:20:06.343796 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:06.343836 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:06.344084 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:06.344762 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:06.344768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:06.345835 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:06.346164 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:06.346184 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.348199 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.348205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:06.349544 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:06.349568 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.351317 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.351324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:06.351588 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:06.352618 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:06.352703 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:06.356963 (+ 4260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1171,"thread_start_us":889,"threads_started":1}
W20260810 03:20:06.357785 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:06.342809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:43847 (local address 127.19.246.254:40097)
0810 03:20:06.343167 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:06.343178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:06.344044 (+ 866us) server_negotiation.cc:407] Connection header received
0810 03:20:06.344250 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:06.344254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:06.344357 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:06.344631 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:06.346421 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.347921 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.349889 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.350980 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.351192 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:06.357385 (+ 6193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:43847: BlockingRecv error: recv got EOF from 127.19.246.194:43847 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Aug 10 03:20:06 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:06 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:06.746030 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:06.733469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:32935)
0810 03:20:06.734124 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:06.734142 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:06.734291 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:06.734765 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:06.734769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:06.735241 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:06.735549 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:06.735561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.737338 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.737348 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:06.739674 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:06.739696 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.740905 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.740911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:06.741067 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:06.741928 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:06.741949 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:06.745625 (+ 3676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":376,"threads_started":1}
W20260810 03:20:06.746459 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:06.733710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:32935 (local address 127.19.246.254:40097)
0810 03:20:06.733909 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:06.733914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:06.734331 (+ 417us) server_negotiation.cc:407] Connection header received
0810 03:20:06.734404 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:06.734408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:06.734478 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:06.734589 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:06.735815 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.737116 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.739903 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.740452 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.740593 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:06.746002 (+ 5409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:32935: BlockingRecv error: recv got EOF from 127.19.246.193:32935 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 03:20:06 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:06 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:06.902024 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:06.887773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:45163)
0810 03:20:06.888812 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:06.888840 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:06.889032 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:06.889824 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:06.889828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:06.890344 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:06.890526 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:06.890536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.892308 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.892314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:06.893416 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:06.893435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.894642 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.894650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:06.894888 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:06.896751 (+ 1863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:06.896789 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:06.901279 (+ 4490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":620,"threads_started":1}
W20260810 03:20:06.902227 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:06.887833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:45163 (local address 127.19.246.254:40097)
0810 03:20:06.888125 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:06.888136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:06.889023 (+ 887us) server_negotiation.cc:407] Connection header received
0810 03:20:06.889474 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:06.889480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:06.889576 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:06.889704 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:06.890790 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.892029 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.893665 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:06.894197 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:06.894336 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:06.901876 (+ 7540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:45163: BlockingRecv error: recv got EOF from 127.19.246.195:45163 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 03:20:07 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:07 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:07.371287 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:07.358756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:51299)
0810 03:20:07.359764 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:07.359791 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:07.359957 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:07.360538 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:07.360542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:07.361062 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:07.361262 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:07.361272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.362706 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.362712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:07.364123 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:07.364143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.365910 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.365917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:07.366280 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:07.367189 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:07.367223 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:07.370955 (+ 3732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":425,"threads_started":1}
W20260810 03:20:07.371778 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:07.359132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:51299 (local address 127.19.246.254:40097)
0810 03:20:07.359474 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:07.359484 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:07.359935 (+ 451us) server_negotiation.cc:407] Connection header received
0810 03:20:07.360166 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:07.360171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:07.360301 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:07.360420 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:07.361445 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.362478 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.364442 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.365446 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.365587 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:07.371477 (+ 5890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:51299: BlockingRecv error: recv got EOF from 127.19.246.194:51299 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Aug 10 03:20:07 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:07 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:07.758821 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:07.747148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:44219)
0810 03:20:07.747737 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:07.747756 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:07.747890 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:07.748213 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:07.748216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:07.748604 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:07.748874 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:07.748885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.750428 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.750433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:07.751736 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:07.751754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.752753 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.752758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:07.752902 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:07.753751 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:07.753772 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:07.758405 (+ 4633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":331,"threads_started":1}
W20260810 03:20:07.758965 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:07.747492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:44219 (local address 127.19.246.254:40097)
0810 03:20:07.747685 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:07.747691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:07.747845 (+ 154us) server_negotiation.cc:407] Connection header received
0810 03:20:07.748003 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:07.748007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:07.748075 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:07.748178 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:07.749081 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.750087 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.751940 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.752497 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.752627 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:07.758665 (+ 6038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:44219: BlockingRecv error: recv got EOF from 127.19.246.193:44219 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 03:20:07 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:07 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:07.916023 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:07.904245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:33779)
0810 03:20:07.904954 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:07.904978 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:07.905206 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:07.906303 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:07.906311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:07.906877 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:07.907149 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:07.907165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.908772 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.908777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:07.909813 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:07.909829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.910756 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.910762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:07.910908 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:07.911891 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:07.911911 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:07.915738 (+ 3827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":391,"threads_started":1}
W20260810 03:20:07.916340 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:07.904371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:33779 (local address 127.19.246.254:40097)
0810 03:20:07.904752 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:07.904760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:07.905528 (+ 768us) server_negotiation.cc:407] Connection header received
0810 03:20:07.905642 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:07.905649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:07.905756 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:07.905872 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:07.907669 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.908603 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.909996 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:07.910474 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:07.910586 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:07.916049 (+ 5463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:33779: BlockingRecv error: recv got EOF from 127.19.246.195:33779 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Aug 10 03:20:08 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:08 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:08.391618 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:08.372778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:47783)
0810 03:20:08.373421 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:08.373455 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:08.373652 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:08.374181 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:08.374186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:08.374686 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:08.374890 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:08.374900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.376520 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.376526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:08.377619 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:08.377637 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.378495 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.378502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:08.378699 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:08.386040 (+ 7341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:08.386073 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:08.390933 (+ 4860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":416,"threads_started":1}
W20260810 03:20:08.391908 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:08.372808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:47783 (local address 127.19.246.254:40097)
0810 03:20:08.373035 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:08.373043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:08.373580 (+ 537us) server_negotiation.cc:407] Connection header received
0810 03:20:08.373900 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:08.373906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:08.373996 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:08.374124 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:08.375070 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.376323 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.377816 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.378288 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.378458 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:08.391459 (+ 13001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:47783: BlockingRecv error: recv got EOF from 127.19.246.194:47783 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 03:20:08 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:08 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.193@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:08.772632 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:08.760628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.193:52759)
0810 03:20:08.761247 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:08.761267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:08.761401 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:08.761864 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:08.761868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:08.762337 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:08.762599 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:08.762611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.763911 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.763916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:08.765070 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:08.765085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.766214 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.766222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:08.766426 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:08.767433 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:08.767453 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:08.772324 (+ 4871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":335,"threads_started":1}
W20260810 03:20:08.772975 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:08.760929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:52759 (local address 127.19.246.254:40097)
0810 03:20:08.761154 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:08.761161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:08.761386 (+ 225us) server_negotiation.cc:407] Connection header received
0810 03:20:08.761605 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:08.761609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:08.761681 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:08.761780 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:08.762854 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.763732 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.765465 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.765990 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.766163 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:08.772661 (+ 6498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.193:52759: BlockingRecv error: recv got EOF from 127.19.246.193:52759 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 03:20:08 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:08 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.195@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:08.934397 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:08.917493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.195:49325)
0810 03:20:08.918389 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:08.918407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:08.918538 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:08.918954 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:08.918959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:08.919472 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:08.919774 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:08.919787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.921261 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.921269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:08.922921 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:08.922962 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.924787 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.924794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:08.924990 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:08.926166 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:08.926207 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:08.933804 (+ 7597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":575,"threads_started":1}
W20260810 03:20:08.934459 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:08.917684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:49325 (local address 127.19.246.254:40097)
0810 03:20:08.917896 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:08.917903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:08.918511 (+ 608us) server_negotiation.cc:407] Connection header received
0810 03:20:08.918647 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:08.918654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:08.918745 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:08.918885 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:08.920008 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.920887 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.923596 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:08.924249 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:08.924665 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:08.934183 (+ 9518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.195:49325: BlockingRecv error: recv got EOF from 127.19.246.195:49325 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 03:20:09 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:20:09 dist-test-slave-btw8 krb5kdc[29511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.246.194@KRBTEST.COM for oryx/127.19.246.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:20:09.404958 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:09.393657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:40097 (local address 127.19.246.194:36167)
0810 03:20:09.394408 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:09.394430 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:09.394600 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:09.394946 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:09.394950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:09.395499 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:09.395683 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:09.395692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.396781 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.396789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:09.397762 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:09.397775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.398558 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.398564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:09.398750 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:09.399879 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:09.399909 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:09.404616 (+ 4707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.246.254:40097: Server oryx/127.19.246.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":349,"threads_started":1}
W20260810 03:20:09.405644 30476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:09.393842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:36167 (local address 127.19.246.254:40097)
0810 03:20:09.394190 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:09.394200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:09.394526 (+ 326us) server_negotiation.cc:407] Connection header received
0810 03:20:09.394733 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:09.394745 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:09.394810 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:09.394918 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:09.395832 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.396611 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.397919 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.398379 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.398527 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:09.405326 (+ 6799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.246.194:36167: BlockingRecv error: recv got EOF from 127.19.246.194:36167 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 03:20:09.708992 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30164
I20260810 03:20:09.729971 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:09.711159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.19.246.193:40163)
0810 03:20:09.711968 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:09.711977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:09.712038 (+ 61us) server_negotiation.cc:407] Connection header received
0810 03:20:09.712275 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:09.712285 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:09.712594 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:09.712874 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:09.715462 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.716573 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.718646 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.719531 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.719689 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:09.723573 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:09.723606 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:09.723627 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:09.723971 (+ 344us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:09.729313 (+ 5342us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.246.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 03:20:09.729703 (+ 390us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34954: Request ticket server kudu/127.19.246.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":448,"threads_started":1}
W20260810 03:20:09.730103 30681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34954: Request ticket server kudu/127.19.246.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 03:20:09.730518 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30164: Not authorized: Client connection negotiation failed: client connection to 127.19.246.193:40163: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.246.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 03:20:09.730772 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 30164
I20260810 03:20:09.758639 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30331
I20260810 03:20:09.776865 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:09.760495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.19.246.194:43077)
0810 03:20:09.761307 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:09.761315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:09.761351 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:20:09.761420 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:09.761427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:09.761681 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:09.761944 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:09.763778 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.765498 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.767340 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.768198 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.768358 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:09.772493 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:09.772505 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:09.772522 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:09.772580 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:09.776196 (+ 3616us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.246.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 03:20:09.776561 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59448: Request ticket server kudu/127.19.246.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":415,"threads_started":1}
W20260810 03:20:09.777027 30682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59448: Request ticket server kudu/127.19.246.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 03:20:09.777316 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30331: Not authorized: Client connection negotiation failed: client connection to 127.19.246.194:43077: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.246.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 03:20:09.777503 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 30331
I20260810 03:20:09.801303 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30478
I20260810 03:20:09.821970 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:09.802774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.19.246.195:35943)
0810 03:20:09.803578 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:09.803585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:09.803614 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:20:09.803681 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:09.803688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:09.804072 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:09.804262 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:09.806217 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.807480 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.809406 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:09.812187 (+ 2781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:09.813800 (+ 1613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:09.817741 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:09.817759 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:09.817782 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:09.817836 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:09.821123 (+ 3287us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.246.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 03:20:09.821686 (+ 563us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39096: Request ticket server kudu/127.19.246.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":345,"threads_started":1}
W20260810 03:20:09.822140 30683 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39096: Request ticket server kudu/127.19.246.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 03:20:09.822724 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30478: Not authorized: Client connection negotiation failed: client connection to 127.19.246.195:35943: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.19.246.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 03:20:09.822893 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 30478
I20260810 03:20:09.848706 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30086
I20260810 03:20:09.850885 30155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:10.014575 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 30086
2026-08-10T03:20:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22804 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T03:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:20:10Z Disabled control of system clock
I20260810 03:20:10.152241 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:42787
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 03:20:10.585122 30688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:10.585489 30688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:10.585652 30688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:10.593693 30688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:20:10.593860 30688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:10.593900 30688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:10.594048 30688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:20:10.594127 30688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:20:10.604911 30688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:42787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:10.606483 30688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:10.608714 30688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:10.627487 30694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:10.627681 30696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:10.627846 30693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:10.628957 30688 server_base.cc:1034] running on GCE node
I20260810 03:20:10.631444 30688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:10.634318 30688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:10.638013 30688 hybrid_clock.cc:648] HybridClock initialized: now 1786332010637934 us; error 109 us; skew 500 ppm
I20260810 03:20:10.638690 30688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:20:10.642514 30688 webserver.cc:460] Webserver started at http://127.19.246.254:38463/ using document root <none> and password file <none>
I20260810 03:20:10.643554 30688 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:10.643733 30688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:10.644127 30688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:10.647733 30688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "497ab502fb3d4f478600d119c249b13f"
format_stamp: "Formatted at 2026-08-10 03:20:10 on dist-test-slave-btw8"
server_key: "2acd683a2599af6db106b96cfb2a456e"
server_key_iv: "eaebcb544d250789c9e09c89909a6a4d"
server_key_version: "encryptionkey@0"
I20260810 03:20:10.648802 30688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "497ab502fb3d4f478600d119c249b13f"
format_stamp: "Formatted at 2026-08-10 03:20:10 on dist-test-slave-btw8"
server_key: "2acd683a2599af6db106b96cfb2a456e"
server_key_iv: "eaebcb544d250789c9e09c89909a6a4d"
server_key_version: "encryptionkey@0"
I20260810 03:20:10.657017 30688 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260810 03:20:10.662295 30703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:10.664513 30688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:20:10.664821 30688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "497ab502fb3d4f478600d119c249b13f"
format_stamp: "Formatted at 2026-08-10 03:20:10 on dist-test-slave-btw8"
server_key: "2acd683a2599af6db106b96cfb2a456e"
server_key_iv: "eaebcb544d250789c9e09c89909a6a4d"
server_key_version: "encryptionkey@0"
I20260810 03:20:10.665166 30688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:10.689756 30688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:10.690997 30688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:10.691471 30688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:10.718699 30688 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:42787
I20260810 03:20:10.718730 30755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:42787 every 8 connection(s)
I20260810 03:20:10.721318 30688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:20:10.723245 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 30688
I20260810 03:20:10.723608 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:20:10.724507 20443 external_mini_cluster.cc:1428] Setting key 00e742100fb385479b2c9346d1006f44
I20260810 03:20:10.728431 30756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:20:10.747913 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:10.727942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.19.246.254:42787)
0810 03:20:10.729681 (+ 1739us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:10.729695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:10.729764 (+ 69us) server_negotiation.cc:407] Connection header received
0810 03:20:10.731932 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:10.731966 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:10.732590 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:10.733009 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:10.734515 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:10.736674 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:10.740236 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:10.741494 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:10.741868 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:10.743916 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:10.743926 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:10.743945 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:10.744666 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:10.744792 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:10.744832 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:10.744961 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:10.745495 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 03:20:10.746021 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":502,"threads_started":1}
I20260810 03:20:10.751976 30756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f: Bootstrap starting.
I20260810 03:20:10.756292 30756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f: Neither blocks nor log segments found. Creating new log.
I20260810 03:20:10.757660 30756 log.cc:824] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:20:10.763794 30756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f: No bootstrap required, opened a new log
I20260810 03:20:10.772858 30756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ab502fb3d4f478600d119c249b13f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42787 } }
I20260810 03:20:10.773378 30756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:20:10.773550 30756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497ab502fb3d4f478600d119c249b13f, State: Initialized, Role: FOLLOWER
I20260810 03:20:10.774912 30756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ab502fb3d4f478600d119c249b13f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42787 } }
I20260810 03:20:10.775202 30756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:20:10.775424 30756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:20:10.775653 30756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:20:10.778282 30756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ab502fb3d4f478600d119c249b13f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42787 } }
I20260810 03:20:10.779019 30756 leader_election.cc:302] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 497ab502fb3d4f478600d119c249b13f; no voters:
I20260810 03:20:10.779841 30756 leader_election.cc:288] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:20:10.780092 30762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:20:10.780997 30762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [term 1 LEADER]: Becoming Leader. State: Replica: 497ab502fb3d4f478600d119c249b13f, State: Running, Role: LEADER
I20260810 03:20:10.781877 30762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ab502fb3d4f478600d119c249b13f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42787 } }
I20260810 03:20:10.782658 30756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:20:10.787418 30764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 497ab502fb3d4f478600d119c249b13f. Latest consensus state: current_term: 1 leader_uuid: "497ab502fb3d4f478600d119c249b13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ab502fb3d4f478600d119c249b13f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42787 } } }
I20260810 03:20:10.788789 30764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [sys.catalog]: This master's current role is: LEADER
I20260810 03:20:10.787603 30763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "497ab502fb3d4f478600d119c249b13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497ab502fb3d4f478600d119c249b13f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 42787 } } }
I20260810 03:20:10.790014 30763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f [sys.catalog]: This master's current role is: LEADER
I20260810 03:20:10.795001 30772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:20:10.801262 30772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:20:10.816641 30772 catalog_manager.cc:1347] Generated new cluster ID: ab055482dec640a3ac83a57c1dd541ff
I20260810 03:20:10.816817 30772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:20:10.832878 30772 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:20:10.834388 30772 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:20:10.849478 30772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 497ab502fb3d4f478600d119c249b13f: Generated new TSK 0
I20260810 03:20:10.851800 30772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:20:10.875581 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42787
--builtin_ntp_servers=127.19.246.212:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 03:20:11.354521 30783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:11.354954 30783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:11.355197 30783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:11.364755 30783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:11.364902 30783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:11.365090 30783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:20:11.378417 30783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:11.381069 30783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:11.384296 30783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:11.405514 30789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:11.405755 30794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:11.406527 30791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:11.407378 30783 server_base.cc:1034] running on GCE node
I20260810 03:20:11.408392 30783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:11.409845 30783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:11.411267 30783 hybrid_clock.cc:648] HybridClock initialized: now 1786332011411245 us; error 127 us; skew 500 ppm
I20260810 03:20:11.411922 30783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:20:11.415521 30783 webserver.cc:460] Webserver started at http://127.19.246.193:46101/ using document root <none> and password file <none>
I20260810 03:20:11.416476 30783 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:11.416669 30783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:11.417126 30783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:11.422189 30783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "642b1fdc30444fd7b028a67eba10f0c9"
format_stamp: "Formatted at 2026-08-10 03:20:11 on dist-test-slave-btw8"
server_key: "e01a8646482e8769719b5cad6b7ce97c"
server_key_iv: "f974bcb9b3a08428c966e9afe06b2887"
server_key_version: "encryptionkey@0"
I20260810 03:20:11.424126 30783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "642b1fdc30444fd7b028a67eba10f0c9"
format_stamp: "Formatted at 2026-08-10 03:20:11 on dist-test-slave-btw8"
server_key: "e01a8646482e8769719b5cad6b7ce97c"
server_key_iv: "f974bcb9b3a08428c966e9afe06b2887"
server_key_version: "encryptionkey@0"
I20260810 03:20:11.433454 30783 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260810 03:20:11.441001 30800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:11.443747 30783 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:20:11.444069 30783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "642b1fdc30444fd7b028a67eba10f0c9"
format_stamp: "Formatted at 2026-08-10 03:20:11 on dist-test-slave-btw8"
server_key: "e01a8646482e8769719b5cad6b7ce97c"
server_key_iv: "f974bcb9b3a08428c966e9afe06b2887"
server_key_version: "encryptionkey@0"
I20260810 03:20:11.444398 30783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:11.477672 30783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:11.479239 30783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:11.480069 30783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:11.481706 30783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:11.484336 30783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:11.484501 30783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:11.484668 30783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:11.484763 30783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:11.535714 30783 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:41877
I20260810 03:20:11.535880 30922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:41877 every 8 connection(s)
I20260810 03:20:11.537427 30783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:20:11.538745 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 30783
I20260810 03:20:11.539057 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:20:11.540082 20443 external_mini_cluster.cc:1428] Setting key ca30ac6c6204ad435bb176874156c356
I20260810 03:20:11.544354 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42787
--builtin_ntp_servers=127.19.246.212:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 03:20:11.559609 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:11.541246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:37773 (local address 127.19.246.254:42787)
0810 03:20:11.541904 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:11.541911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:11.543944 (+ 2033us) server_negotiation.cc:407] Connection header received
0810 03:20:11.547804 (+ 3860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:11.547810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:11.547901 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:11.548018 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:11.550395 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:11.551540 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:11.555123 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:11.555753 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:11.555907 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:11.558343 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:11.558350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:11.558354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:11.558381 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:11.558437 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:11.558455 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:11.558605 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:11.558924 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 03:20:11.559078 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":320,"threads_started":1}
I20260810 03:20:11.561053 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:11.542189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42787 (local address 127.19.246.193:37773)
0810 03:20:11.543700 (+ 1511us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:11.543765 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:11.547215 (+ 3450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:11.548513 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:11.548532 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:11.548713 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:11.550006 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:11.550046 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:11.551777 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:11.551788 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:11.554730 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:11.554767 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:11.556141 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:11.556155 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:11.556639 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:11.557799 (+ 1160us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:20:11.557804 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:11.557874 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:20:11.558003 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:11.558607 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:11.558625 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:11.558638 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 03:20:11.558913 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 03:20:11.559528 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":403,"threads_started":1}
I20260810 03:20:11.563642 30923 heartbeater.cc:347] Connected to a master server at 127.19.246.254:42787
I20260810 03:20:11.564311 30923 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:11.565887 30923 heartbeater.cc:511] Master 127.19.246.254:42787 requested a full tablet report, sending...
I20260810 03:20:11.569159 30720 ts_manager.cc:194] Registered new tserver with Master: 642b1fdc30444fd7b028a67eba10f0c9 (127.19.246.193:41877)
I20260810 03:20:11.571475 30720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.193:37773
W20260810 03:20:11.989881 30927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:11.990319 30927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:11.990495 30927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:11.999918 30927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:12.000130 30927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:12.000317 30927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:20:12.014623 30927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:12.016556 30927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:12.018769 30927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:12.036816 30932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:12.037039 30935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:12.037593 30933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:12.038942 30927 server_base.cc:1034] running on GCE node
I20260810 03:20:12.040025 30927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:12.041268 30927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:12.042583 30927 hybrid_clock.cc:648] HybridClock initialized: now 1786332012042455 us; error 121 us; skew 500 ppm
I20260810 03:20:12.043004 30927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:20:12.046499 30927 webserver.cc:460] Webserver started at http://127.19.246.194:46357/ using document root <none> and password file <none>
I20260810 03:20:12.047531 30927 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:12.047705 30927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:12.048092 30927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:12.051859 30927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "4ad852cb06f54472bd04110188b5dc7e"
format_stamp: "Formatted at 2026-08-10 03:20:12 on dist-test-slave-btw8"
server_key: "724f28a10115f678442454138efaed14"
server_key_iv: "b5659a852f58c298d4d73d1bec31b4cc"
server_key_version: "encryptionkey@0"
I20260810 03:20:12.052896 30927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "4ad852cb06f54472bd04110188b5dc7e"
format_stamp: "Formatted at 2026-08-10 03:20:12 on dist-test-slave-btw8"
server_key: "724f28a10115f678442454138efaed14"
server_key_iv: "b5659a852f58c298d4d73d1bec31b4cc"
server_key_version: "encryptionkey@0"
I20260810 03:20:12.064301 30927 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.004s
I20260810 03:20:12.071336 30942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:12.073734 30927 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260810 03:20:12.074045 30927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "4ad852cb06f54472bd04110188b5dc7e"
format_stamp: "Formatted at 2026-08-10 03:20:12 on dist-test-slave-btw8"
server_key: "724f28a10115f678442454138efaed14"
server_key_iv: "b5659a852f58c298d4d73d1bec31b4cc"
server_key_version: "encryptionkey@0"
I20260810 03:20:12.074854 30927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:12.101227 30927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:12.102412 30927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:12.102851 30927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:12.104665 30927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:12.106670 30927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:12.106864 30927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:12.107030 30927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:12.107126 30927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:12.153162 30927 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:36727
I20260810 03:20:12.153218 31063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:36727 every 8 connection(s)
I20260810 03:20:12.154878 30927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:20:12.158942 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 30927
I20260810 03:20:12.159183 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:20:12.160342 20443 external_mini_cluster.cc:1428] Setting key 5865028b2b3fdc526e0e7e39a4d0c73e
I20260810 03:20:12.164582 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42787
--builtin_ntp_servers=127.19.246.212:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 03:20:12.173904 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:12.158516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:38473 (local address 127.19.246.254:42787)
0810 03:20:12.159190 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:12.159198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:12.161140 (+ 1942us) server_negotiation.cc:407] Connection header received
0810 03:20:12.164613 (+ 3473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:12.164619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:12.164725 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:12.164843 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:12.166631 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.167632 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.170002 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.170546 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.170708 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:12.172764 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:12.172770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:12.172774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:12.172797 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:12.172844 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:12.172859 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:12.172979 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:12.173317 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 03:20:12.173451 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":354,"threads_started":1}
I20260810 03:20:12.175339 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:12.159506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42787 (local address 127.19.246.194:38473)
0810 03:20:12.160917 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:12.160968 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:12.163949 (+ 2981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:12.165207 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:12.165222 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:12.165357 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:12.166324 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:12.166351 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.167867 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.167872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:12.169746 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:12.169776 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.170957 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.170964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:12.171220 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:12.172256 (+ 1036us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:20:12.172260 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:12.172322 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:20:12.172434 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:12.172981 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:12.172996 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:12.173008 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:20:12.173306 (+ 298us) client_negotiation.cc:240] Negotiation successful
0810 03:20:12.174020 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":559,"threads_started":1}
I20260810 03:20:12.177731 31064 heartbeater.cc:347] Connected to a master server at 127.19.246.254:42787
I20260810 03:20:12.178296 31064 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:12.179746 31064 heartbeater.cc:511] Master 127.19.246.254:42787 requested a full tablet report, sending...
I20260810 03:20:12.182699 30720 ts_manager.cc:194] Registered new tserver with Master: 4ad852cb06f54472bd04110188b5dc7e (127.19.246.194:36727)
I20260810 03:20:12.183826 30720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.194:38473
I20260810 03:20:12.576390 30923 heartbeater.cc:503] Master 127.19.246.254:42787 was elected leader, sending a full tablet report...
W20260810 03:20:12.585420 31068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:12.585785 31068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:12.585914 31068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:12.594179 31068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:12.594317 31068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:12.594486 31068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:20:12.606536 31068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:42787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:12.608268 31068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:12.610366 31068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:12.628844 31075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:12.628736 31077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:12.628881 31074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:12.629642 31068 server_base.cc:1034] running on GCE node
I20260810 03:20:12.630626 31068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:12.632011 31068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:12.633260 31068 hybrid_clock.cc:648] HybridClock initialized: now 1786332012633218 us; error 54 us; skew 500 ppm
I20260810 03:20:12.633684 31068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:20:12.636793 31068 webserver.cc:460] Webserver started at http://127.19.246.195:45865/ using document root <none> and password file <none>
I20260810 03:20:12.637764 31068 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:12.637899 31068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:12.638335 31068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:12.642179 31068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "9afab14aae6a4e65a22aa89dea420073"
format_stamp: "Formatted at 2026-08-10 03:20:12 on dist-test-slave-btw8"
server_key: "fea1a4fa92663bca12c53e6ca122f949"
server_key_iv: "50f5d500940d31bc31ec9bd51fc63efc"
server_key_version: "encryptionkey@0"
I20260810 03:20:12.643435 31068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "9afab14aae6a4e65a22aa89dea420073"
format_stamp: "Formatted at 2026-08-10 03:20:12 on dist-test-slave-btw8"
server_key: "fea1a4fa92663bca12c53e6ca122f949"
server_key_iv: "50f5d500940d31bc31ec9bd51fc63efc"
server_key_version: "encryptionkey@0"
I20260810 03:20:12.650722 31068 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260810 03:20:12.655839 31084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:12.658236 31068 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260810 03:20:12.658450 31068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "9afab14aae6a4e65a22aa89dea420073"
format_stamp: "Formatted at 2026-08-10 03:20:12 on dist-test-slave-btw8"
server_key: "fea1a4fa92663bca12c53e6ca122f949"
server_key_iv: "50f5d500940d31bc31ec9bd51fc63efc"
server_key_version: "encryptionkey@0"
I20260810 03:20:12.658813 31068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:12.678931 31068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:12.680055 31068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:12.680413 31068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:12.681773 31068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:12.683770 31068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:12.683879 31068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:12.684062 31068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:12.684150 31068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:12.725812 31068 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:34359
I20260810 03:20:12.725835 31209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:34359 every 8 connection(s)
I20260810 03:20:12.727432 31068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:20:12.736368 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 31068
I20260810 03:20:12.736639 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:20:12.737360 20443 external_mini_cluster.cc:1428] Setting key d48b8ed0b84c11e038ef14468b08d363
I20260810 03:20:12.744020 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:12.731027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:40813 (local address 127.19.246.254:42787)
0810 03:20:12.731805 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:12.731811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:12.733496 (+ 1685us) server_negotiation.cc:407] Connection header received
0810 03:20:12.735759 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:12.735766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:12.735895 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:12.736059 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:12.737570 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.738423 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.740526 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.740994 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.741284 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:12.742979 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:12.742984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:12.742988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:12.743007 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:12.743059 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:12.743073 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:12.743183 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:12.743489 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 03:20:12.743614 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":451,"threads_started":1}
I20260810 03:20:12.745175 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:12.731876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:42787 (local address 127.19.246.195:40813)
0810 03:20:12.733264 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:12.733319 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:12.735247 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:12.736362 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:12.736372 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:12.736530 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:12.737323 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:12.737361 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.738638 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.738643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:12.740323 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:12.740341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.741415 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.741423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:12.741710 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:12.742569 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:20:12.742572 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:12.742619 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:20:12.742720 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:12.743185 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:12.743199 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:12.743210 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 03:20:12.743480 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 03:20:12.743976 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":603,"threads_started":1}
I20260810 03:20:12.746891 31211 heartbeater.cc:347] Connected to a master server at 127.19.246.254:42787
I20260810 03:20:12.747337 31211 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:12.748312 31211 heartbeater.cc:511] Master 127.19.246.254:42787 requested a full tablet report, sending...
I20260810 03:20:12.750618 30720 ts_manager.cc:194] Registered new tserver with Master: 9afab14aae6a4e65a22aa89dea420073 (127.19.246.195:34359)
I20260810 03:20:12.751710 30720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.195:40813
I20260810 03:20:12.755429 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:20:12.764642 31214 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:12.762346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.19.246.254:42787)
0810 03:20:12.762641 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:12.762646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:12.763220 (+ 574us) server_negotiation.cc:407] Connection header received
0810 03:20:12.763448 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:12.763453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:12.763530 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:12.763768 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:12.764320 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34200: BlockingRecv error: recv got EOF from 127.0.0.1:34200 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
I20260810 03:20:12.765340 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30783
I20260810 03:20:12.776103 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:12.766801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.19.246.193:41877)
0810 03:20:12.767591 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:12.767599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:12.767630 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:20:12.767695 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:12.767702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:12.767942 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:12.768122 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:12.768709 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.769918 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.771780 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.772469 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.772603 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:12.774203 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:12.774214 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:12.774231 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:12.774956 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:12.775013 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:12.775043 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:12.775161 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:12.775526 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 03:20:12.775746 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":395,"threads_started":1}
I20260810 03:20:12.777681 30918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:12.871625 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 30783
I20260810 03:20:12.893213 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30927
I20260810 03:20:12.903739 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:12.894422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.19.246.194:36727)
0810 03:20:12.895492 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:12.895500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:12.895533 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:20:12.895611 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:12.895621 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:12.895895 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:12.896130 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:12.896633 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.897699 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.899084 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:12.899943 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:12.900170 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:12.901564 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:12.901574 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:12.901590 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:12.902289 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:12.902469 (+ 180us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:12.902501 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:12.902877 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:12.903062 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 03:20:12.903265 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":578,"threads_started":1}
I20260810 03:20:12.905722 31059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:12.996456 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 30927
I20260810 03:20:13.020709 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 31068
I20260810 03:20:13.032821 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:13.022053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.19.246.195:34359)
0810 03:20:13.022940 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:13.022947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:13.022983 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:20:13.023058 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:13.023066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:13.023337 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:13.023779 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:13.024389 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:13.025680 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:13.027732 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:13.029034 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:13.029237 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:13.030795 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:13.030807 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:13.030824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:13.031623 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:13.031690 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:13.031727 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:13.031840 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:13.032106 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 03:20:13.032371 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":446,"threads_started":1}
I20260810 03:20:13.034349 31205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:13.136611 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 31068
I20260810 03:20:13.158829 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 30688
I20260810 03:20:13.160313 30750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:13.277311 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 30688
2026-08-10T03:20:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3241 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T03:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:20:13Z Disabled control of system clock
I20260810 03:20:13.381461 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:39235
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:39235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:20:13.813704 31233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:13.814081 31233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:13.814182 31233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:13.823654 31233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:20:13.823827 31233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:13.823879 31233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:13.823949 31233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:20:13.824002 31233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:20:13.835634 31233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:39235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:39235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:13.837107 31233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:13.839174 31233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:13.854178 31238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:13.854223 31242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:13.854260 31239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:13.855720 31233 server_base.cc:1034] running on GCE node
I20260810 03:20:13.856964 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:13.858958 31233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:13.860298 31233 hybrid_clock.cc:648] HybridClock initialized: now 1786332013860251 us; error 74 us; skew 500 ppm
I20260810 03:20:13.860787 31233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:20:13.869022 31233 webserver.cc:460] Webserver started at http://127.19.246.254:32899/ using document root <none> and password file <none>
I20260810 03:20:13.869971 31233 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:13.870146 31233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:13.870544 31233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:13.874655 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "bcf447f22fa54d17ac846e4d5d56bb61"
format_stamp: "Formatted at 2026-08-10 03:20:13 on dist-test-slave-btw8"
server_key: "093266e53b7f90563caad2bb1d41a5b8"
server_key_iv: "c8ff7e43447639e752b89698a1d261b9"
server_key_version: "encryptionkey@0"
I20260810 03:20:13.875943 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "bcf447f22fa54d17ac846e4d5d56bb61"
format_stamp: "Formatted at 2026-08-10 03:20:13 on dist-test-slave-btw8"
server_key: "093266e53b7f90563caad2bb1d41a5b8"
server_key_iv: "c8ff7e43447639e752b89698a1d261b9"
server_key_version: "encryptionkey@0"
I20260810 03:20:13.883144 31233 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260810 03:20:13.888253 31248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:13.890257 31233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:20:13.890584 31233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "bcf447f22fa54d17ac846e4d5d56bb61"
format_stamp: "Formatted at 2026-08-10 03:20:13 on dist-test-slave-btw8"
server_key: "093266e53b7f90563caad2bb1d41a5b8"
server_key_iv: "c8ff7e43447639e752b89698a1d261b9"
server_key_version: "encryptionkey@0"
I20260810 03:20:13.890913 31233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:13.899863 31233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:13.900959 31233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:13.901360 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:13.931762 31233 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:39235
I20260810 03:20:13.931811 31304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:39235 every 8 connection(s)
I20260810 03:20:13.933630 31233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:20:13.938571 31306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:20:13.942534 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 31233
I20260810 03:20:13.942927 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:20:13.943854 20443 external_mini_cluster.cc:1428] Setting key 23184ccf1155ba7c1680f891376b8f92
I20260810 03:20:13.954586 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:13.946635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.19.246.254:39235)
0810 03:20:13.947959 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:13.947972 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:13.948021 (+ 49us) server_negotiation.cc:407] Connection header received
0810 03:20:13.949963 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:13.949994 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:13.950613 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:13.951076 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:13.952540 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:13.952548 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:13.952564 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:13.952587 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:13.952664 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:13.952695 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:13.952925 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:13.953049 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 03:20:13.953462 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":496,"threads_started":1}
I20260810 03:20:13.954988 31306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61: Bootstrap starting.
I20260810 03:20:13.959136 31306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61: Neither blocks nor log segments found. Creating new log.
I20260810 03:20:13.960552 31306 log.cc:824] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61: Log is configured to *not* fsync() on all Append() calls
I20260810 03:20:13.966105 31306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61: No bootstrap required, opened a new log
I20260810 03:20:13.975878 31306 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf447f22fa54d17ac846e4d5d56bb61" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39235 } }
I20260810 03:20:13.976442 31306 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:20:13.976586 31306 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf447f22fa54d17ac846e4d5d56bb61, State: Initialized, Role: FOLLOWER
I20260810 03:20:13.977332 31306 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf447f22fa54d17ac846e4d5d56bb61" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39235 } }
I20260810 03:20:13.977654 31306 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:20:13.977849 31306 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:20:13.978068 31306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:20:13.980902 31306 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf447f22fa54d17ac846e4d5d56bb61" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39235 } }
I20260810 03:20:13.981647 31306 leader_election.cc:302] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcf447f22fa54d17ac846e4d5d56bb61; no voters:
I20260810 03:20:13.982534 31306 leader_election.cc:288] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:20:13.982929 31313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:20:13.984246 31313 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [term 1 LEADER]: Becoming Leader. State: Replica: bcf447f22fa54d17ac846e4d5d56bb61, State: Running, Role: LEADER
I20260810 03:20:13.985141 31313 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf447f22fa54d17ac846e4d5d56bb61" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39235 } }
I20260810 03:20:13.986130 31306 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:20:13.989472 31315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcf447f22fa54d17ac846e4d5d56bb61. Latest consensus state: current_term: 1 leader_uuid: "bcf447f22fa54d17ac846e4d5d56bb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf447f22fa54d17ac846e4d5d56bb61" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39235 } } }
I20260810 03:20:13.990223 31315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [sys.catalog]: This master's current role is: LEADER
I20260810 03:20:13.989472 31314 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcf447f22fa54d17ac846e4d5d56bb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf447f22fa54d17ac846e4d5d56bb61" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 39235 } } }
I20260810 03:20:13.990460 31314 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61 [sys.catalog]: This master's current role is: LEADER
I20260810 03:20:13.993345 31319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:20:14.001319 31319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:20:14.014160 31319 catalog_manager.cc:1347] Generated new cluster ID: d13857a06bc5490a9a3bf1b24561fd9e
I20260810 03:20:14.014331 31319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:20:14.032037 31319 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:20:14.032667 31319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:20:14.048228 31319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcf447f22fa54d17ac846e4d5d56bb61: Generated new TSK 0
I20260810 03:20:14.049639 31319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:20:14.064147 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39235
--builtin_ntp_servers=127.19.246.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:20:14.514402 31332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:14.514770 31332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:14.514876 31332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:14.524825 31332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:14.524987 31332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:14.525120 31332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:20:14.537371 31332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:14.539006 31332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:14.541733 31332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:14.558727 31338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:14.558780 31342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:14.558809 31340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:14.560619 31332 server_base.cc:1034] running on GCE node
I20260810 03:20:14.561528 31332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:14.562774 31332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:14.564112 31332 hybrid_clock.cc:648] HybridClock initialized: now 1786332014564041 us; error 54 us; skew 500 ppm
I20260810 03:20:14.564550 31332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:20:14.568704 31332 webserver.cc:460] Webserver started at http://127.19.246.193:34057/ using document root <none> and password file <none>
I20260810 03:20:14.569940 31332 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:14.570230 31332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:14.570721 31332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:14.574507 31332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "1dec235da68c4c95a3a311bb84609644"
format_stamp: "Formatted at 2026-08-10 03:20:14 on dist-test-slave-btw8"
server_key: "6ad3cb20c238fe4d6045569a62d10386"
server_key_iv: "b7f2af89f4d0235e230d1738ae83676e"
server_key_version: "encryptionkey@0"
I20260810 03:20:14.575762 31332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "1dec235da68c4c95a3a311bb84609644"
format_stamp: "Formatted at 2026-08-10 03:20:14 on dist-test-slave-btw8"
server_key: "6ad3cb20c238fe4d6045569a62d10386"
server_key_iv: "b7f2af89f4d0235e230d1738ae83676e"
server_key_version: "encryptionkey@0"
I20260810 03:20:14.582588 31332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260810 03:20:14.587972 31350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:14.589988 31332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:20:14.590317 31332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "1dec235da68c4c95a3a311bb84609644"
format_stamp: "Formatted at 2026-08-10 03:20:14 on dist-test-slave-btw8"
server_key: "6ad3cb20c238fe4d6045569a62d10386"
server_key_iv: "b7f2af89f4d0235e230d1738ae83676e"
server_key_version: "encryptionkey@0"
I20260810 03:20:14.590682 31332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:14.598516 31332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:14.599742 31332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:14.600126 31332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:14.601387 31332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:14.603427 31332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:14.603624 31332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 03:20:14.603791 31332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:14.603900 31332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:14.649389 31332 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:39089
I20260810 03:20:14.649441 31468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:39089 every 8 connection(s)
I20260810 03:20:14.651156 31332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:20:14.658432 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 31332
I20260810 03:20:14.658903 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:20:14.660053 20443 external_mini_cluster.cc:1428] Setting key 40f9e10ae812d4674a6f7cb048fb29ac
I20260810 03:20:14.662564 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:14.654457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:34111 (local address 127.19.246.254:39235)
0810 03:20:14.655062 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:14.655068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:14.656535 (+ 1467us) server_negotiation.cc:407] Connection header received
0810 03:20:14.658815 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:14.658822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:14.658921 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:14.659075 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:14.661100 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:14.661108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:14.661113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:14.661143 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:14.661231 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:14.661255 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:14.661438 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:14.662089 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 03:20:14.662178 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":342,"threads_started":1}
I20260810 03:20:14.664000 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:14.655032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39235 (local address 127.19.246.193:34111)
0810 03:20:14.656305 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:14.656355 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:14.658315 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:14.659477 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:14.659491 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:14.659617 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:14.660538 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:20:14.660542 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:14.660610 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:20:14.660769 (+ 159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:14.661616 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:14.661632 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:14.661643 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 03:20:14.662113 (+ 470us) client_negotiation.cc:240] Negotiation successful
0810 03:20:14.662421 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":435,"threads_started":1}
I20260810 03:20:14.665401 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39235
--builtin_ntp_servers=127.19.246.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 03:20:14.666687 31469 heartbeater.cc:347] Connected to a master server at 127.19.246.254:39235
I20260810 03:20:14.667078 31469 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:14.667879 31469 heartbeater.cc:511] Master 127.19.246.254:39235 requested a full tablet report, sending...
I20260810 03:20:14.670483 31266 ts_manager.cc:194] Registered new tserver with Master: 1dec235da68c4c95a3a311bb84609644 (127.19.246.193:39089)
W20260810 03:20:15.082093 31474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:15.082451 31474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:15.082552 31474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:15.091005 31474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:15.091152 31474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:15.091295 31474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:20:15.102157 31474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:15.103709 31474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:15.105705 31474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:15.127815 31482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:15.127807 31480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:15.128798 31479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:15.130298 31474 server_base.cc:1034] running on GCE node
I20260810 03:20:15.131271 31474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:15.132642 31474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:15.133917 31474 hybrid_clock.cc:648] HybridClock initialized: now 1786332015133869 us; error 47 us; skew 500 ppm
I20260810 03:20:15.134485 31474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:20:15.143994 31474 webserver.cc:460] Webserver started at http://127.19.246.194:43957/ using document root <none> and password file <none>
I20260810 03:20:15.146015 31474 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:15.146219 31474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:15.146735 31474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:15.151280 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "1ddb14d341b74133b05c0894a117a5eb"
format_stamp: "Formatted at 2026-08-10 03:20:15 on dist-test-slave-btw8"
server_key: "f959f3cf83cb8a4c91202a176fb03cf3"
server_key_iv: "fd6c7454f114544f3269efd7e492f92d"
server_key_version: "encryptionkey@0"
I20260810 03:20:15.152546 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "1ddb14d341b74133b05c0894a117a5eb"
format_stamp: "Formatted at 2026-08-10 03:20:15 on dist-test-slave-btw8"
server_key: "f959f3cf83cb8a4c91202a176fb03cf3"
server_key_iv: "fd6c7454f114544f3269efd7e492f92d"
server_key_version: "encryptionkey@0"
I20260810 03:20:15.159679 31474 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 03:20:15.165586 31489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:15.167558 31474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:20:15.167831 31474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "1ddb14d341b74133b05c0894a117a5eb"
format_stamp: "Formatted at 2026-08-10 03:20:15 on dist-test-slave-btw8"
server_key: "f959f3cf83cb8a4c91202a176fb03cf3"
server_key_iv: "fd6c7454f114544f3269efd7e492f92d"
server_key_version: "encryptionkey@0"
I20260810 03:20:15.168134 31474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:15.177124 31474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:15.178437 31474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:15.178845 31474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:15.180373 31474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:15.182351 31474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:15.182521 31474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:15.182696 31474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:15.182802 31474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:15.241072 31474 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:45621
I20260810 03:20:15.241084 31611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:45621 every 8 connection(s)
I20260810 03:20:15.243054 31474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:20:15.250325 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 31474
I20260810 03:20:15.250546 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:20:15.251513 20443 external_mini_cluster.cc:1428] Setting key d373d9e5a9e1a066bb0a003d459a16d9
I20260810 03:20:15.256979 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39235
--builtin_ntp_servers=127.19.246.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 03:20:15.262223 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:15.248392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:50423 (local address 127.19.246.254:39235)
0810 03:20:15.249327 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:15.249332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:15.252996 (+ 3664us) server_negotiation.cc:407] Connection header received
0810 03:20:15.255045 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:15.255051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:15.257581 (+ 2530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:15.257876 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:15.260868 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:15.260886 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:15.260896 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:15.260942 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:15.261052 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:15.261086 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:15.261453 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:15.261679 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 03:20:15.261781 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":554,"threads_started":1}
I20260810 03:20:15.263923 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:15.250061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39235 (local address 127.19.246.194:50423)
0810 03:20:15.252403 (+ 2342us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:15.252469 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:15.254485 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:15.258368 (+ 3883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:15.258389 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:15.258584 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:15.260055 (+ 1471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:20:15.260062 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:15.260149 (+ 87us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:20:15.260313 (+ 164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:15.261324 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:15.261340 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:15.261354 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 03:20:15.261783 (+ 429us) client_negotiation.cc:240] Negotiation successful
0810 03:20:15.262175 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"thread_start_us":1149,"threads_started":1}
I20260810 03:20:15.266661 31612 heartbeater.cc:347] Connected to a master server at 127.19.246.254:39235
I20260810 03:20:15.267411 31612 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:15.268934 31612 heartbeater.cc:511] Master 127.19.246.254:39235 requested a full tablet report, sending...
I20260810 03:20:15.271636 31266 ts_manager.cc:194] Registered new tserver with Master: 1ddb14d341b74133b05c0894a117a5eb (127.19.246.194:45621)
I20260810 03:20:15.674434 31469 heartbeater.cc:503] Master 127.19.246.254:39235 was elected leader, sending a full tablet report...
W20260810 03:20:15.738692 31616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:15.739298 31616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:15.739545 31616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:15.750840 31616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:15.751060 31616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:15.751286 31616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:20:15.763600 31616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:39235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:15.765187 31616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:15.768471 31616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:15.787838 31622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:15.787879 31624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:15.787838 31621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:15.790750 31616 server_base.cc:1034] running on GCE node
I20260810 03:20:15.791708 31616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:15.793169 31616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:15.795305 31616 hybrid_clock.cc:648] HybridClock initialized: now 1786332015795156 us; error 143 us; skew 500 ppm
I20260810 03:20:15.795876 31616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:20:15.800091 31616 webserver.cc:460] Webserver started at http://127.19.246.195:36785/ using document root <none> and password file <none>
I20260810 03:20:15.801079 31616 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:15.801195 31616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:15.801590 31616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:15.805502 31616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "085161d74ce64d0ea50ad39a7eca3a1f"
format_stamp: "Formatted at 2026-08-10 03:20:15 on dist-test-slave-btw8"
server_key: "67903ebb204c81af1538fa5676ac2e61"
server_key_iv: "fc95566c6490b6dda76c66090e96bd82"
server_key_version: "encryptionkey@0"
I20260810 03:20:15.807142 31616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "085161d74ce64d0ea50ad39a7eca3a1f"
format_stamp: "Formatted at 2026-08-10 03:20:15 on dist-test-slave-btw8"
server_key: "67903ebb204c81af1538fa5676ac2e61"
server_key_iv: "fc95566c6490b6dda76c66090e96bd82"
server_key_version: "encryptionkey@0"
I20260810 03:20:15.816699 31616 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260810 03:20:15.821884 31630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:15.824116 31616 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 03:20:15.824368 31616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "085161d74ce64d0ea50ad39a7eca3a1f"
format_stamp: "Formatted at 2026-08-10 03:20:15 on dist-test-slave-btw8"
server_key: "67903ebb204c81af1538fa5676ac2e61"
server_key_iv: "fc95566c6490b6dda76c66090e96bd82"
server_key_version: "encryptionkey@0"
I20260810 03:20:15.824746 31616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:15.832603 31616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:15.833827 31616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:15.834280 31616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:15.835664 31616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:15.837807 31616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:15.837936 31616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:15.838131 31616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:15.838215 31616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:15.882658 31616 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:37381
I20260810 03:20:15.882721 31756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:37381 every 8 connection(s)
I20260810 03:20:15.884461 31616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:20:15.894119 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 31616
I20260810 03:20:15.894448 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:20:15.895076 20443 external_mini_cluster.cc:1428] Setting key 4dba14910a66ab853f12d07c5c86044b
I20260810 03:20:15.898620 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:15.888806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:33683 (local address 127.19.246.254:39235)
0810 03:20:15.889417 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:15.889423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:15.892054 (+ 2631us) server_negotiation.cc:407] Connection header received
0810 03:20:15.895464 (+ 3410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:15.895470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:15.895553 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:15.895717 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:15.897639 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:15.897644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:15.897648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:15.897669 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:15.897723 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:15.897739 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:15.897888 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:15.898147 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 03:20:15.898230 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":322,"threads_started":1}
I20260810 03:20:15.900274 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:15.889956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:39235 (local address 127.19.246.195:33683)
0810 03:20:15.891815 (+ 1859us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:15.891874 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:15.894869 (+ 2995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:15.896065 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:15.896077 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:15.896206 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:15.897199 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:20:15.897202 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:15.897253 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:20:15.897367 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:15.897855 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:15.897868 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:15.897877 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:20:15.898271 (+ 394us) client_negotiation.cc:240] Negotiation successful
0810 03:20:15.898612 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":596,"threads_started":1}
I20260810 03:20:15.902786 31757 heartbeater.cc:347] Connected to a master server at 127.19.246.254:39235
I20260810 03:20:15.903990 31757 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:15.905234 31757 heartbeater.cc:511] Master 127.19.246.254:39235 requested a full tablet report, sending...
I20260810 03:20:15.908377 31266 ts_manager.cc:194] Registered new tserver with Master: 085161d74ce64d0ea50ad39a7eca3a1f (127.19.246.195:37381)
I20260810 03:20:15.915606 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:20:15.925046 31759 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:20:15.921756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.19.246.254:39235)
0810 03:20:15.922058 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:15.922069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:15.922764 (+ 695us) server_negotiation.cc:407] Connection header received
0810 03:20:15.922937 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:15.922942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:15.923063 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:15.923178 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:15.924765 (+ 1587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52712: BlockingRecv error: recv got EOF from 127.0.0.1:52712 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 03:20:15.926539 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 31332
I20260810 03:20:15.932631 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:15.928075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.19.246.193:39089)
0810 03:20:15.929289 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:15.929301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:15.929345 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:20:15.929579 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:15.929590 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:15.930014 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:15.930214 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:15.931704 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:15.931719 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:15.931733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:15.931752 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:15.931808 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:15.931825 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:15.932029 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:15.932109 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 03:20:15.932239 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":615,"threads_started":1}
I20260810 03:20:15.935698 31464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:16.031576 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 31332
I20260810 03:20:16.055805 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 31474
I20260810 03:20:16.061810 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:16.057155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.19.246.194:45621)
0810 03:20:16.057754 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:16.057762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:16.057794 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:20:16.057876 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:16.057883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:16.058132 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:16.058296 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:16.060290 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:16.060303 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:16.060322 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:16.060350 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:16.060456 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:16.060490 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:16.060678 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:16.061240 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 03:20:16.061428 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":299,"threads_started":1}
I20260810 03:20:16.063992 31606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:16.158923 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 31474
I20260810 03:20:16.180012 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 31616
I20260810 03:20:16.184700 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:16.181375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.19.246.195:37381)
0810 03:20:16.182068 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:16.182076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:16.182109 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:20:16.182181 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:16.182188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:16.182446 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:16.182593 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:16.183670 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:16.183677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:16.183692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:20:16.183718 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:16.183763 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:20:16.183778 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:16.183928 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:16.184075 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 03:20:16.184168 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":354,"threads_started":1}
I20260810 03:20:16.186393 31750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:16.278057 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 31616
I20260810 03:20:16.300282 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 31233
I20260810 03:20:16.302491 31300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:16.427515 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 31233
2026-08-10T03:20:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (3151 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:20:16 dist-test-slave-btw8 krb5kdc[31777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:20:16 dist-test-slave-btw8 krb5kdc[31777](info): set up 2 sockets
Aug 10 03:20:16 dist-test-slave-btw8 krb5kdc[31777](info): commencing operation
krb5kdc: starting...
W20260810 03:20:17.874935 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.393s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:20:18 dist-test-slave-btw8 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:20:18.278196 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36349
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:20:18.711285 31792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:18.711760 31792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:18.711938 31792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:18.720789 31792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:20:18.721026 31792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:18.721156 31792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:18.721273 31792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:20:18.721364 31792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:20:18.733521 31792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:18.735167 31792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:18.737129 31792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:18.755808 31797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:18.755927 31800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:18.755954 31798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:18.757117 31792 server_base.cc:1034] running on GCE node
I20260810 03:20:18.758860 31792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:18.761651 31792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:18.763200 31792 hybrid_clock.cc:648] HybridClock initialized: now 1786332018763081 us; error 100 us; skew 500 ppm
I20260810 03:20:18.763890 31792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:20:18 dist-test-slave-btw8 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332018, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:20:18.769698 31792 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:20:18.772028 31792 webserver.cc:460] Webserver started at http://127.19.246.254:40245/ using document root <none> and password file <none>
I20260810 03:20:18.772941 31792 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:18.773159 31792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:18.773563 31792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:18.778918 31792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "7540332657b9472b9f71bbbaa4bdf588"
format_stamp: "Formatted at 2026-08-10 03:20:18 on dist-test-slave-btw8"
server_key: "fb9b5173acf1f3ce57f5a60227b011a1"
server_key_iv: "d1a1c6a0547bdadb2f72cdc4b7b7cdab"
server_key_version: "encryptionkey@0"
I20260810 03:20:18.781024 31792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "7540332657b9472b9f71bbbaa4bdf588"
format_stamp: "Formatted at 2026-08-10 03:20:18 on dist-test-slave-btw8"
server_key: "fb9b5173acf1f3ce57f5a60227b011a1"
server_key_iv: "d1a1c6a0547bdadb2f72cdc4b7b7cdab"
server_key_version: "encryptionkey@0"
I20260810 03:20:18.790782 31792 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.007s
I20260810 03:20:18.797219 31807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:18.800103 31792 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:20:18.800405 31792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "7540332657b9472b9f71bbbaa4bdf588"
format_stamp: "Formatted at 2026-08-10 03:20:18 on dist-test-slave-btw8"
server_key: "fb9b5173acf1f3ce57f5a60227b011a1"
server_key_iv: "d1a1c6a0547bdadb2f72cdc4b7b7cdab"
server_key_version: "encryptionkey@0"
I20260810 03:20:18.800690 31792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:18.823163 31792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:18.830819 31792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:18.831687 31792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:18.862645 31792 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:36349
I20260810 03:20:18.862689 31860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:36349 every 8 connection(s)
I20260810 03:20:18.864516 31792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:20:18.869417 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:20:18.870812 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 31792
I20260810 03:20:18.871096 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:20:18.872107 20443 external_mini_cluster.cc:1428] Setting key d1b17b5986dbd9e47ddf8c280d9a3b8b
I20260810 03:20:18.886127 31861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588: Bootstrap starting.
I20260810 03:20:18.890363 31861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588: Neither blocks nor log segments found. Creating new log.
I20260810 03:20:18.891873 31861 log.cc:824] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588: Log is configured to *not* fsync() on all Append() calls
I20260810 03:20:18.896060 31861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588: No bootstrap required, opened a new log
Aug 10 03:20:18 dist-test-slave-btw8 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:20:18.907255 31861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540332657b9472b9f71bbbaa4bdf588" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36349 } }
I20260810 03:20:18.907975 31861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:20:18.908106 31861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7540332657b9472b9f71bbbaa4bdf588, State: Initialized, Role: FOLLOWER
I20260810 03:20:18.909525 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:18.874859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.19.246.254:36349)
0810 03:20:18.876282 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:18.876304 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:18.876404 (+ 100us) server_negotiation.cc:407] Connection header received
0810 03:20:18.879233 (+ 2829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:18.879271 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:18.880009 (+ 738us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:18.880569 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:18.883117 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:18.885432 (+ 2315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:18.887335 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:18.888340 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:18.888636 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:18.898553 (+ 9917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:18.898566 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:18.898589 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:18.898647 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:18.903012 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:18.904134 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:18.904143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:18.904152 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:18.904432 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:18.905087 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:18.905094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:18.905108 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:18.906076 (+ 968us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:18.906628 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:18.907463 (+ 835us) server_negotiation.cc:299] Negotiation successful
0810 03:20:18.908132 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":619,"threads_started":1}
I20260810 03:20:18.909405 31861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540332657b9472b9f71bbbaa4bdf588" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36349 } }
I20260810 03:20:18.909834 31861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:20:18.910039 31861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:20:18.910542 31861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:20:18.914294 31861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540332657b9472b9f71bbbaa4bdf588" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36349 } }
I20260810 03:20:18.915058 31861 leader_election.cc:302] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7540332657b9472b9f71bbbaa4bdf588; no voters:
I20260810 03:20:18.915900 31861 leader_election.cc:288] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:20:18.916110 31866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:20:18.917260 31866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [term 1 LEADER]: Becoming Leader. State: Replica: 7540332657b9472b9f71bbbaa4bdf588, State: Running, Role: LEADER
I20260810 03:20:18.918180 31866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540332657b9472b9f71bbbaa4bdf588" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36349 } }
I20260810 03:20:18.918900 31861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:20:18.922610 31868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7540332657b9472b9f71bbbaa4bdf588. Latest consensus state: current_term: 1 leader_uuid: "7540332657b9472b9f71bbbaa4bdf588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540332657b9472b9f71bbbaa4bdf588" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36349 } } }
I20260810 03:20:18.922566 31867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7540332657b9472b9f71bbbaa4bdf588" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7540332657b9472b9f71bbbaa4bdf588" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36349 } } }
I20260810 03:20:18.923170 31867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [sys.catalog]: This master's current role is: LEADER
I20260810 03:20:18.923170 31868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588 [sys.catalog]: This master's current role is: LEADER
I20260810 03:20:18.924984 31872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:20:18.934482 31872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:20:18.949882 31872 catalog_manager.cc:1347] Generated new cluster ID: c104478701a642e296f0080eae6638b5
I20260810 03:20:18.950048 31872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:20:18.968636 31872 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:20:18.970229 31872 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:20:18.978662 31872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588: Generated new TSK 0
I20260810 03:20:18.980105 31872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:20:19.073097 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36349
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:20:19.563061 31887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:19.563582 31887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:19.563843 31887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:19.573042 31887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:19.573211 31887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:19.573342 31887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:20:19.584795 31887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:19.586393 31887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:19.588150 31887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:19.609803 31892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:19.609865 31893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:19.609905 31895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:19.611639 31887 server_base.cc:1034] running on GCE node
I20260810 03:20:19.612560 31887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:19.613857 31887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:19.615126 31887 hybrid_clock.cc:648] HybridClock initialized: now 1786332019615053 us; error 74 us; skew 500 ppm
I20260810 03:20:19.615764 31887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:20:19 dist-test-slave-btw8 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332019, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:20:19.622232 31887 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:20:19.624577 31887 webserver.cc:460] Webserver started at http://127.19.246.193:38249/ using document root <none> and password file <none>
I20260810 03:20:19.625444 31887 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:19.625597 31887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:19.625954 31887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:19.630254 31887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "9e4a63a1cf03419488cb1a68eb20c649"
format_stamp: "Formatted at 2026-08-10 03:20:19 on dist-test-slave-btw8"
server_key: "973cc976c9f20681249c1c33de6ba969"
server_key_iv: "e383c02e2f244a98dbb0048f9f92261b"
server_key_version: "encryptionkey@0"
I20260810 03:20:19.631340 31887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "9e4a63a1cf03419488cb1a68eb20c649"
format_stamp: "Formatted at 2026-08-10 03:20:19 on dist-test-slave-btw8"
server_key: "973cc976c9f20681249c1c33de6ba969"
server_key_iv: "e383c02e2f244a98dbb0048f9f92261b"
server_key_version: "encryptionkey@0"
I20260810 03:20:19.639792 31887 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260810 03:20:19.644817 31903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:19.647087 31887 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:20:19.647296 31887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "9e4a63a1cf03419488cb1a68eb20c649"
format_stamp: "Formatted at 2026-08-10 03:20:19 on dist-test-slave-btw8"
server_key: "973cc976c9f20681249c1c33de6ba969"
server_key_iv: "e383c02e2f244a98dbb0048f9f92261b"
server_key_version: "encryptionkey@0"
I20260810 03:20:19.647619 31887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:19.679322 31887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:19.686146 31887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:19.686486 31887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:19.687872 31887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:19.689659 31887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:19.689755 31887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:19.689854 31887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:19.689893 31887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:19.738206 31887 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:35901
I20260810 03:20:19.738274 32031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:35901 every 8 connection(s)
I20260810 03:20:19.739813 31887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:20:19.744756 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 31887
I20260810 03:20:19.745064 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:20:19.746587 20443 external_mini_cluster.cc:1428] Setting key bd16e35ce3d82cab0eb63619f4418343
Aug 10 03:20:19 dist-test-slave-btw8 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332019, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:20:19.771636 32035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:19.743423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:41629 (local address 127.19.246.254:36349)
0810 03:20:19.744039 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:19.744045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:19.745649 (+ 1604us) server_negotiation.cc:407] Connection header received
0810 03:20:19.748328 (+ 2679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:19.748335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:19.748446 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:19.748595 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:19.751539 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:19.752663 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:19.757222 (+ 4559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:19.757727 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:19.758118 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:19.763728 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:19.763734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:19.763738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:19.763784 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:19.767648 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:19.768713 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:19.768718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:19.768720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:19.768877 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:19.769428 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:19.769432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:19.769435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:19.769808 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:19.769992 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:19.770845 (+ 853us) server_negotiation.cc:299] Negotiation successful
0810 03:20:19.771177 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":339,"threads_started":1}
I20260810 03:20:19.772821 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:19.744308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36349 (local address 127.19.246.193:41629)
0810 03:20:19.745358 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:19.745419 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:19.747808 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:19.748990 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:19.749001 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:19.750030 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:19.751145 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:19.751187 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:19.752925 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:19.752935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:19.756831 (+ 3896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:19.756870 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:19.758119 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:19.758130 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:19.758598 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:19.759696 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:19.759737 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:19.763313 (+ 3576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:19.767921 (+ 4608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:20:19.767931 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:20:19.767948 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:20:19.768526 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:20:19.769013 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:20:19.769019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:20:19.769023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:20:19.769267 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:20:19.769965 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:19.769974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:19.770406 (+ 432us) client_negotiation.cc:769] Sending connection context
0810 03:20:19.770740 (+ 334us) client_negotiation.cc:240] Negotiation successful
0810 03:20:19.771440 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":452,"threads_started":1}
I20260810 03:20:19.774935 32032 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36349
I20260810 03:20:19.775516 32032 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:19.776839 32032 heartbeater.cc:511] Master 127.19.246.254:36349 requested a full tablet report, sending...
I20260810 03:20:19.780298 31824 ts_manager.cc:194] Registered new tserver with Master: 9e4a63a1cf03419488cb1a68eb20c649 (127.19.246.193:35901)
I20260810 03:20:19.783265 31824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:41629
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:20:19.840725 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36349
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:20:20.349926 32040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:20.350330 32040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:20.350446 32040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:20.360982 32040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:20.361157 32040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:20.361805 32040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:20:20.375794 32040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:20.377429 32040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:20.378974 32040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:20.397943 32045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:20.397943 32046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:20.398356 32048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:20.399811 32040 server_base.cc:1034] running on GCE node
I20260810 03:20:20.400674 32040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:20.402277 32040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:20.403766 32040 hybrid_clock.cc:648] HybridClock initialized: now 1786332020403699 us; error 59 us; skew 500 ppm
I20260810 03:20:20.404227 32040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:20:20 dist-test-slave-btw8 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332020, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:20:20.409597 32040 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:20:20.412150 32040 webserver.cc:460] Webserver started at http://127.19.246.194:42593/ using document root <none> and password file <none>
I20260810 03:20:20.413133 32040 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:20.413331 32040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:20.413712 32040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:20.418263 32040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "499217140e464c7a9a9f4c571b26c5c1"
format_stamp: "Formatted at 2026-08-10 03:20:20 on dist-test-slave-btw8"
server_key: "e8dfe4c50f287889cfcc98bf1ef69ca4"
server_key_iv: "bbce03bc36a5915b95ffdc37d72a3843"
server_key_version: "encryptionkey@0"
I20260810 03:20:20.420056 32040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "499217140e464c7a9a9f4c571b26c5c1"
format_stamp: "Formatted at 2026-08-10 03:20:20 on dist-test-slave-btw8"
server_key: "e8dfe4c50f287889cfcc98bf1ef69ca4"
server_key_iv: "bbce03bc36a5915b95ffdc37d72a3843"
server_key_version: "encryptionkey@0"
I20260810 03:20:20.430282 32040 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.008s
I20260810 03:20:20.435738 32055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:20.437985 32040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260810 03:20:20.438846 32040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "499217140e464c7a9a9f4c571b26c5c1"
format_stamp: "Formatted at 2026-08-10 03:20:20 on dist-test-slave-btw8"
server_key: "e8dfe4c50f287889cfcc98bf1ef69ca4"
server_key_iv: "bbce03bc36a5915b95ffdc37d72a3843"
server_key_version: "encryptionkey@0"
I20260810 03:20:20.439242 32040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:20.460023 32040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:20.466192 32040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:20.466769 32040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:20.468171 32040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:20.469990 32040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:20.470145 32040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:20.470351 32040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:20.470450 32040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:20.526021 32040 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:40887
I20260810 03:20:20.526501 32182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:40887 every 8 connection(s)
I20260810 03:20:20.528182 32040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:20:20.537901 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 32040
I20260810 03:20:20.538257 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:20:20.538833 20443 external_mini_cluster.cc:1428] Setting key c2f5ceef250252a3e5e6b29534dcb68e
Aug 10 03:20:20 dist-test-slave-btw8 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332020, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:20:20.561983 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:20.533306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:44869 (local address 127.19.246.254:36349)
0810 03:20:20.534195 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:20.534204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:20.536220 (+ 2016us) server_negotiation.cc:407] Connection header received
0810 03:20:20.539499 (+ 3279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:20.539507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:20.539640 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:20.539794 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:20.541927 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:20.542825 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:20.545036 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:20.545555 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:20.545739 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:20.552040 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:20.552053 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:20.552062 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:20.552128 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:20.557041 (+ 4913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:20.558631 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:20.558637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:20.558639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:20.558838 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:20.559575 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:20.559580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:20.559583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:20.559963 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:20.560121 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:20.561331 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0810 03:20:20.561602 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":448,"threads_started":1}
I20260810 03:20:20.563894 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:20.534449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36349 (local address 127.19.246.194:44869)
0810 03:20:20.535923 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:20.535995 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:20.538977 (+ 2982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:20.540165 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:20.540178 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:20.540899 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:20.541678 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:20.541704 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:20.543063 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:20.543069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:20.544756 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:20.544776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:20.545889 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:20.545898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:20.546248 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:20.547226 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:20.547255 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:20.551110 (+ 3855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:20.557358 (+ 6248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:20:20.557375 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:20:20.557401 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:20:20.558331 (+ 930us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:20:20.558994 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:20:20.559003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:20:20.559011 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:20:20.559392 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:20:20.560136 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:20.560149 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:20.560755 (+ 606us) client_negotiation.cc:769] Sending connection context
0810 03:20:20.561157 (+ 402us) client_negotiation.cc:240] Negotiation successful
0810 03:20:20.561947 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"thread_start_us":573,"threads_started":1}
I20260810 03:20:20.566543 32183 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36349
I20260810 03:20:20.567188 32183 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:20.568935 32183 heartbeater.cc:511] Master 127.19.246.254:36349 requested a full tablet report, sending...
I20260810 03:20:20.571995 31824 ts_manager.cc:194] Registered new tserver with Master: 499217140e464c7a9a9f4c571b26c5c1 (127.19.246.194:40887)
I20260810 03:20:20.573135 31824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:44869
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:20:20.629106 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36349
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:20:20.789145 32032 heartbeater.cc:503] Master 127.19.246.254:36349 was elected leader, sending a full tablet report...
W20260810 03:20:21.083997 32189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:20:21.084456 32189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:20:21.084708 32189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:20:21.094403 32189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:20:21.094663 32189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:20:21.094877 32189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:20:21.108150 32189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:20:21.111037 32189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:20:21.113903 32189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:20:21.136004 32195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:21.136101 32197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:20:21.136128 32194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:20:21.139263 32189 server_base.cc:1034] running on GCE node
I20260810 03:20:21.140339 32189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:20:21.141769 32189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:20:21.143220 32189 hybrid_clock.cc:648] HybridClock initialized: now 1786332021143145 us; error 61 us; skew 500 ppm
I20260810 03:20:21.144114 32189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:20:21 dist-test-slave-btw8 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332021, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:20:21.152814 32189 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:20:21.155728 32189 webserver.cc:460] Webserver started at http://127.19.246.195:38403/ using document root <none> and password file <none>
I20260810 03:20:21.156654 32189 fs_manager.cc:359] Metadata directory not provided
I20260810 03:20:21.156810 32189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:20:21.157186 32189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:20:21.160840 32189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca"
format_stamp: "Formatted at 2026-08-10 03:20:21 on dist-test-slave-btw8"
server_key: "3437476db0008e1d3859508a83418b3e"
server_key_iv: "adc616de4bd3fa25bddc0c12d152a1d1"
server_key_version: "encryptionkey@0"
I20260810 03:20:21.162029 32189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca"
format_stamp: "Formatted at 2026-08-10 03:20:21 on dist-test-slave-btw8"
server_key: "3437476db0008e1d3859508a83418b3e"
server_key_iv: "adc616de4bd3fa25bddc0c12d152a1d1"
server_key_version: "encryptionkey@0"
I20260810 03:20:21.171876 32189 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260810 03:20:21.180511 32205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:21.183815 32189 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260810 03:20:21.184779 32189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca"
format_stamp: "Formatted at 2026-08-10 03:20:21 on dist-test-slave-btw8"
server_key: "3437476db0008e1d3859508a83418b3e"
server_key_iv: "adc616de4bd3fa25bddc0c12d152a1d1"
server_key_version: "encryptionkey@0"
I20260810 03:20:21.185312 32189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:20:21.230968 32189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:20:21.237970 32189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:20:21.238413 32189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:20:21.239724 32189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:20:21.241482 32189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:20:21.241576 32189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:21.241783 32189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:20:21.241860 32189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:20:21.289219 32189 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:43037
I20260810 03:20:21.289402 32334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:43037 every 8 connection(s)
I20260810 03:20:21.290778 32189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:20:21.295032 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 32189
I20260810 03:20:21.295269 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:20:21.296001 20443 external_mini_cluster.cc:1428] Setting key 1e1d6d479a2aa43712737aa0a96ba114
Aug 10 03:20:21 dist-test-slave-btw8 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332021, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:20:21.322202 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.296089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:43951 (local address 127.19.246.254:36349)
0810 03:20:21.297045 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.297053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.298010 (+ 957us) server_negotiation.cc:407] Connection header received
0810 03:20:21.300993 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.300998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.301132 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.301260 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:21.303169 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.304050 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.306044 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.307766 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.307968 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.313156 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:21.313165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:21.313176 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:21.313233 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:21.316882 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:21.318508 (+ 1626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:21.318527 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:21.318529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:21.318758 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:21.319920 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:21.319924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:21.319927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:21.320352 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:21.320498 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.321566 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.321841 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":612,"threads_started":1}
I20260810 03:20:21.323729 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.296619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36349 (local address 127.19.246.195:43951)
0810 03:20:21.297807 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:21.297876 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:21.300617 (+ 2741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:21.301558 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:21.301569 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:21.302220 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:20:21.302921 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.302944 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.304313 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.304318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.305742 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.305760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.308137 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.308151 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.308433 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.309267 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:20:21.309301 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:20:21.312862 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:20:21.317220 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:20:21.317231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:20:21.317256 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:20:21.317876 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:20:21.319038 (+ 1162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:20:21.319049 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:20:21.319053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:20:21.319565 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:20:21.320543 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:20:21.320551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:20:21.321099 (+ 548us) client_negotiation.cc:769] Sending connection context
0810 03:20:21.321403 (+ 304us) client_negotiation.cc:240] Negotiation successful
0810 03:20:21.321890 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":496,"threads_started":1}
I20260810 03:20:21.326610 32335 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36349
I20260810 03:20:21.327173 32335 heartbeater.cc:464] Registering TS with master...
I20260810 03:20:21.328298 32335 heartbeater.cc:511] Master 127.19.246.254:36349 requested a full tablet report, sending...
I20260810 03:20:21.331598 31824 ts_manager.cc:194] Registered new tserver with Master: dc7f3794774d4bdba7a8e47ed7a8b2ca (127.19.246.195:43037)
I20260810 03:20:21.333354 31824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:43951
I20260810 03:20:21.344588 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:20:21.371021 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.350812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.19.246.254:36349)
0810 03:20:21.351113 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.351118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.352261 (+ 1143us) server_negotiation.cc:407] Connection header received
0810 03:20:21.352341 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.352346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.352447 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.352554 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:21.355589 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.356408 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.358021 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.358634 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.358863 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.362088 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:21.362101 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:21.362104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:21.362153 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:21.366252 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:21.367256 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:21.367262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:21.367268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:21.367541 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:21.368324 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:21.368330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:21.368333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:21.368758 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:21.369068 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.369496 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.370212 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260810 03:20:21.379671 31824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:20:21.390398 31824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:20:21.433990 32351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.413057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:40887 (local address 127.0.0.1:58586)
0810 03:20:21.416652 (+ 3595us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:21.416705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:21.416973 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:21.418491 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:21.418504 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:21.418557 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:20:21.419317 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.419341 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.422674 (+ 3333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.422684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.426956 (+ 4272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.426993 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.431250 (+ 4257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.431433 (+ 183us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.432287 (+ 854us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.432388 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 03:20:21.432576 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 03:20:21.432882 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3355,"mutex_wait_us":1,"thread_start_us":2835,"threads_started":1}
I20260810 03:20:21.434382 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.412954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.19.246.193:35901)
0810 03:20:21.415305 (+ 2351us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.415318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.418552 (+ 3234us) server_negotiation.cc:407] Connection header received
0810 03:20:21.418716 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.418729 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.419093 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.419506 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:20:21.420991 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.425520 (+ 4529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.429182 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.431811 (+ 2629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.432361 (+ 550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.432656 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.432777 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.433029 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2083,"thread_start_us":1917,"threads_started":1}
I20260810 03:20:21.434479 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.413655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.19.246.194:40887)
0810 03:20:21.414628 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.414644 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.416885 (+ 2241us) server_negotiation.cc:407] Connection header received
0810 03:20:21.417309 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.417320 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.417719 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.417991 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:20:21.419699 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.422311 (+ 2612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.427647 (+ 5336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.430387 (+ 2740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.430731 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.430977 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.433790 (+ 2813us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.434040 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":585,"threads_started":1}
I20260810 03:20:21.434522 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.412438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:35901 (local address 127.0.0.1:55154)
0810 03:20:21.418135 (+ 5697us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:21.418162 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:21.418335 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:21.419822 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:21.419831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:21.419875 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:20:21.420684 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.420704 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.425866 (+ 5162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.425876 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.428969 (+ 3093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.429015 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.432271 (+ 3256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.432279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.432525 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.432564 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:20:21.432678 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:20:21.432900 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3766,"mutex_wait_us":7,"thread_start_us":3740,"threads_started":1}
I20260810 03:20:21.440059 31955 tablet_service.cc:1467] Processing CreateTablet for tablet bb420e3f81004059ab16afe998c885d8 (DEFAULT_TABLE table=test-table [id=4cc1713cd33c4b7d9c203431aabde0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:20:21.440372 32107 tablet_service.cc:1467] Processing CreateTablet for tablet bb420e3f81004059ab16afe998c885d8 (DEFAULT_TABLE table=test-table [id=4cc1713cd33c4b7d9c203431aabde0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:20:21.441978 31955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb420e3f81004059ab16afe998c885d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:20:21.442561 32355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.417271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:43037 (local address 127.0.0.1:47448)
0810 03:20:21.428484 (+ 11213us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:21.428510 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:21.428676 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:21.430470 (+ 1794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:21.430627 (+ 157us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:21.430670 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:20:21.431209 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.431227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.434266 (+ 3039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.434273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.438449 (+ 4176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.438486 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.441543 (+ 3057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.441552 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.441786 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.441830 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:20:21.441956 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 03:20:21.442162 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11057,"thread_start_us":11629,"threads_started":1}
I20260810 03:20:21.442849 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.417289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.19.246.195:43037)
0810 03:20:21.427770 (+ 10481us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.427781 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.429224 (+ 1443us) server_negotiation.cc:407] Connection header received
0810 03:20:21.429375 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.429385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.429994 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.430268 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:20:21.431922 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.433999 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.438819 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.441163 (+ 2344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.441475 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.441721 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.442075 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.442297 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10267,"thread_start_us":9811,"threads_started":1}
I20260810 03:20:21.443176 32107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb420e3f81004059ab16afe998c885d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:20:21.452303 32255 tablet_service.cc:1467] Processing CreateTablet for tablet bb420e3f81004059ab16afe998c885d8 (DEFAULT_TABLE table=test-table [id=4cc1713cd33c4b7d9c203431aabde0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:20:21.455670 32255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb420e3f81004059ab16afe998c885d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:20:21.463078 32357 tablet_bootstrap.cc:492] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: Bootstrap starting.
I20260810 03:20:21.467049 32358 tablet_bootstrap.cc:492] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Bootstrap starting.
I20260810 03:20:21.471423 32358 tablet_bootstrap.cc:654] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Neither blocks nor log segments found. Creating new log.
I20260810 03:20:21.472740 32358 log.cc:824] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:20:21.473989 32357 tablet_bootstrap.cc:654] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: Neither blocks nor log segments found. Creating new log.
I20260810 03:20:21.475785 32357 log.cc:824] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: Log is configured to *not* fsync() on all Append() calls
I20260810 03:20:21.478471 32358 tablet_bootstrap.cc:492] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: No bootstrap required, opened a new log
I20260810 03:20:21.478474 32361 tablet_bootstrap.cc:492] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Bootstrap starting.
I20260810 03:20:21.479077 32358 ts_tablet_manager.cc:1397] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260810 03:20:21.482323 32357 tablet_bootstrap.cc:492] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: No bootstrap required, opened a new log
I20260810 03:20:21.482991 32357 ts_tablet_manager.cc:1397] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.000s
I20260810 03:20:21.483891 32361 tablet_bootstrap.cc:654] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Neither blocks nor log segments found. Creating new log.
I20260810 03:20:21.485939 32361 log.cc:824] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Log is configured to *not* fsync() on all Append() calls
I20260810 03:20:21.493083 32361 tablet_bootstrap.cc:492] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: No bootstrap required, opened a new log
I20260810 03:20:21.493883 32361 ts_tablet_manager.cc:1397] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260810 03:20:21.495183 32358 raft_consensus.cc:348] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
I20260810 03:20:21.496198 32358 raft_consensus.cc:374] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:20:21.496371 32358 raft_consensus.cc:729] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499217140e464c7a9a9f4c571b26c5c1, State: Initialized, Role: FOLLOWER
I20260810 03:20:21.497475 32358 consensus_queue.cc:260] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
I20260810 03:20:21.498845 32357 raft_consensus.cc:348] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
I20260810 03:20:21.499758 32357 raft_consensus.cc:374] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:20:21.499929 32357 raft_consensus.cc:729] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a63a1cf03419488cb1a68eb20c649, State: Initialized, Role: FOLLOWER
I20260810 03:20:21.500563 32183 heartbeater.cc:503] Master 127.19.246.254:36349 was elected leader, sending a full tablet report...
I20260810 03:20:21.500874 32357 consensus_queue.cc:260] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
I20260810 03:20:21.501410 32358 ts_tablet_manager.cc:1428] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Time spent starting tablet: real 0.022s user 0.015s sys 0.005s
I20260810 03:20:21.504616 32357 ts_tablet_manager.cc:1428] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: Time spent starting tablet: real 0.021s user 0.022s sys 0.000s
I20260810 03:20:21.508342 32361 raft_consensus.cc:348] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
I20260810 03:20:21.509150 32361 raft_consensus.cc:374] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:20:21.509294 32361 raft_consensus.cc:729] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7f3794774d4bdba7a8e47ed7a8b2ca, State: Initialized, Role: FOLLOWER
I20260810 03:20:21.510356 32361 consensus_queue.cc:260] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
I20260810 03:20:21.512023 32335 heartbeater.cc:503] Master 127.19.246.254:36349 was elected leader, sending a full tablet report...
I20260810 03:20:21.512457 32361 ts_tablet_manager.cc:1428] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
W20260810 03:20:21.531893 32184 tablet.cc:2369] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:20:21.541919 32363 raft_consensus.cc:482] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:20:21.542570 32363 raft_consensus.cc:504] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
W20260810 03:20:21.544134 32336 tablet.cc:2369] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:20:21.546490 32363 leader_election.cc:288] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e4a63a1cf03419488cb1a68eb20c649 (127.19.246.193:35901), dc7f3794774d4bdba7a8e47ed7a8b2ca (127.19.246.195:43037)
I20260810 03:20:21.557163 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.547166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:60511 (local address 127.19.246.193:35901)
0810 03:20:21.547472 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.547479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.547902 (+ 423us) server_negotiation.cc:407] Connection header received
0810 03:20:21.548219 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.548225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.548314 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.548393 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:20:21.549364 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.550566 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.554813 (+ 4247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.556014 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.556278 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.556373 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.556719 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.556865 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 03:20:21.557271 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.546783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:35901 (local address 127.19.246.194:60511)
0810 03:20:21.547742 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:21.547771 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:21.547985 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:21.548647 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:21.548651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:21.548674 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:20:21.549096 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.549109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.550902 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.550909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.554363 (+ 3454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.554386 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.556321 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.556329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.556517 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.556546 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 03:20:21.556673 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 03:20:21.556846 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":421,"threads_started":1}
I20260810 03:20:21.558565 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.547848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:43037 (local address 127.19.246.194:55195)
0810 03:20:21.548618 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 03:20:21.548635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:20:21.548764 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:20:21.549363 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:20:21.549368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:20:21.549400 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:20:21.549985 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.549999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.552046 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.552051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.555286 (+ 3235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:20:21.555310 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.557278 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.557289 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:20:21.557825 (+ 536us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.557880 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 03:20:21.558031 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 03:20:21.558185 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":401,"threads_started":1}
I20260810 03:20:21.558667 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.548201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:55195 (local address 127.19.246.195:43037)
0810 03:20:21.548444 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.548452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.548737 (+ 285us) server_negotiation.cc:407] Connection header received
0810 03:20:21.548884 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.548891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.548976 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.549086 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:20:21.550290 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.551661 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.555670 (+ 4009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.556856 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.557161 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.557279 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.558104 (+ 825us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.558393 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 03:20:21.558663 31977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb420e3f81004059ab16afe998c885d8" candidate_uuid: "499217140e464c7a9a9f4c571b26c5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a63a1cf03419488cb1a68eb20c649" is_pre_election: true
I20260810 03:20:21.559470 31977 raft_consensus.cc:2454] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 499217140e464c7a9a9f4c571b26c5c1 in term 0.
I20260810 03:20:21.560478 32060 leader_election.cc:302] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499217140e464c7a9a9f4c571b26c5c1, 9e4a63a1cf03419488cb1a68eb20c649; no voters:
I20260810 03:20:21.560642 32279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb420e3f81004059ab16afe998c885d8" candidate_uuid: "499217140e464c7a9a9f4c571b26c5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" is_pre_election: true
I20260810 03:20:21.560943 32363 raft_consensus.cc:2781] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:20:21.561069 32363 raft_consensus.cc:482] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:20:21.561141 32363 raft_consensus.cc:3037] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:20:21.561189 32279 raft_consensus.cc:2454] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 499217140e464c7a9a9f4c571b26c5c1 in term 0.
I20260810 03:20:21.564728 32363 raft_consensus.cc:504] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
I20260810 03:20:21.565603 32363 leader_election.cc:288] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9e4a63a1cf03419488cb1a68eb20c649 (127.19.246.193:35901), dc7f3794774d4bdba7a8e47ed7a8b2ca (127.19.246.195:43037)
I20260810 03:20:21.566430 32279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb420e3f81004059ab16afe998c885d8" candidate_uuid: "499217140e464c7a9a9f4c571b26c5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca"
I20260810 03:20:21.566550 31977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb420e3f81004059ab16afe998c885d8" candidate_uuid: "499217140e464c7a9a9f4c571b26c5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a63a1cf03419488cb1a68eb20c649"
I20260810 03:20:21.566742 32279 raft_consensus.cc:3037] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:20:21.566810 31977 raft_consensus.cc:3037] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:20:21.570849 32279 raft_consensus.cc:2454] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 499217140e464c7a9a9f4c571b26c5c1 in term 1.
I20260810 03:20:21.570849 31977 raft_consensus.cc:2454] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 499217140e464c7a9a9f4c571b26c5c1 in term 1.
I20260810 03:20:21.571636 32060 leader_election.cc:302] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499217140e464c7a9a9f4c571b26c5c1, 9e4a63a1cf03419488cb1a68eb20c649; no voters:
I20260810 03:20:21.572399 32363 raft_consensus.cc:2781] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:20:21.573348 32363 raft_consensus.cc:686] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 499217140e464c7a9a9f4c571b26c5c1, State: Running, Role: LEADER
I20260810 03:20:21.573987 32363 consensus_queue.cc:237] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } }
I20260810 03:20:21.581729 31824 catalog_manager.cc:5495] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 499217140e464c7a9a9f4c571b26c5c1 (127.19.246.194). New cstate: current_term: 1 leader_uuid: "499217140e464c7a9a9f4c571b26c5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "499217140e464c7a9a9f4c571b26c5c1" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 40887 } health_report { overall_health: HEALTHY } } }
I20260810 03:20:21.636776 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.627854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.19.246.194:40887)
0810 03:20:21.628543 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.628555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.628595 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:20:21.628734 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.628738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.628842 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.629068 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:20:21.629967 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.630762 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.632677 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.633225 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.633374 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.634598 (+ 1224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:20:21.635504 (+ 906us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:20:21.635712 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.636065 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.636218 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260810 03:20:21.649641 32279 raft_consensus.cc:1261] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 1 FOLLOWER]: Refusing update from remote peer 499217140e464c7a9a9f4c571b26c5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:20:21.649641 31977 raft_consensus.cc:1261] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 1 FOLLOWER]: Refusing update from remote peer 499217140e464c7a9a9f4c571b26c5c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:20:21.651813 32369 consensus_queue.cc:1035] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a63a1cf03419488cb1a68eb20c649" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 35901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:20:21.652521 32363 consensus_queue.cc:1035] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc7f3794774d4bdba7a8e47ed7a8b2ca" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:20:21.690063 31823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:20:21.690498 31823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:20:21.696731 31823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7540332657b9472b9f71bbbaa4bdf588: Sending DeleteTablet for 3 replicas of tablet bb420e3f81004059ab16afe998c885d8
I20260810 03:20:21.698510 32255 tablet_service.cc:1514] Processing DeleteTablet for tablet bb420e3f81004059ab16afe998c885d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:20:21 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:47448
I20260810 03:20:21.698657 32107 tablet_service.cc:1514] Processing DeleteTablet for tablet bb420e3f81004059ab16afe998c885d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:20:21 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:58586
I20260810 03:20:21.698940 31955 tablet_service.cc:1514] Processing DeleteTablet for tablet bb420e3f81004059ab16afe998c885d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:20:21 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:55154
I20260810 03:20:21.702809 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 31887
I20260810 03:20:21.712913 32385 tablet_replica.cc:330] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: stopping tablet replica
I20260810 03:20:21.713814 32385 raft_consensus.cc:2229] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:20:21.714685 32385 raft_consensus.cc:2258] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:20:21.718207 32385 ts_tablet_manager.cc:1905] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:20:21.729411 32385 ts_tablet_manager.cc:1918] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:20:21.729800 32385 log.cc:1196] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/bb420e3f81004059ab16afe998c885d8
I20260810 03:20:21.730574 32385 ts_tablet_manager.cc:1939] T bb420e3f81004059ab16afe998c885d8 P 9e4a63a1cf03419488cb1a68eb20c649: Deleting consensus metadata
I20260810 03:20:21.733091 31811 catalog_manager.cc:4841] TS 9e4a63a1cf03419488cb1a68eb20c649 (127.19.246.193:35901): tablet bb420e3f81004059ab16afe998c885d8 (table test-table [id=4cc1713cd33c4b7d9c203431aabde0e2]) successfully deleted
I20260810 03:20:21.736047 32383 tablet_replica.cc:330] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: stopping tablet replica
I20260810 03:20:21.736616 32383 raft_consensus.cc:2229] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:20:21.737226 32383 raft_consensus.cc:2258] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:20:21.738225 32384 tablet_replica.cc:330] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: stopping tablet replica
I20260810 03:20:21.738809 32384 raft_consensus.cc:2229] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:20:21.740223 32384 raft_consensus.cc:2258] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:20:21.741868 32383 ts_tablet_manager.cc:1905] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 03:20:21 dist-test-slave-btw8 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:20:21.746546 32384 ts_tablet_manager.cc:1905] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:20:21.751335 32383 ts_tablet_manager.cc:1918] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:20:21.751668 32383 log.cc:1196] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/bb420e3f81004059ab16afe998c885d8
I20260810 03:20:21.752277 32383 ts_tablet_manager.cc:1939] T bb420e3f81004059ab16afe998c885d8 P dc7f3794774d4bdba7a8e47ed7a8b2ca: Deleting consensus metadata
I20260810 03:20:21.754685 31811 catalog_manager.cc:4841] TS dc7f3794774d4bdba7a8e47ed7a8b2ca (127.19.246.195:43037): tablet bb420e3f81004059ab16afe998c885d8 (table test-table [id=4cc1713cd33c4b7d9c203431aabde0e2]) successfully deleted
I20260810 03:20:21.758972 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.705434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.19.246.193:35901)
0810 03:20:21.705746 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.705756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.728535 (+ 22779us) server_negotiation.cc:407] Connection header received
0810 03:20:21.729179 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.729185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.729274 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.729358 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:21.731549 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.732653 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.734773 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.735720 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.735862 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.747609 (+ 11747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:21.747635 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:21.747659 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:21.747729 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:21.753022 (+ 5293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:21.754664 (+ 1642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:21.754676 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:21.754680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:21.754906 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:21.755544 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:21.755551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:21.755554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:21.756416 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:21.757174 (+ 758us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.758028 (+ 854us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.758369 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260810 03:20:21.761497 32027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:21.766031 32384 ts_tablet_manager.cc:1918] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:20:21.766494 32384 log.cc:1196] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/bb420e3f81004059ab16afe998c885d8
I20260810 03:20:21.767834 32384 ts_tablet_manager.cc:1939] T bb420e3f81004059ab16afe998c885d8 P 499217140e464c7a9a9f4c571b26c5c1: Deleting consensus metadata
I20260810 03:20:21.770814 31809 catalog_manager.cc:4841] TS 499217140e464c7a9a9f4c571b26c5c1 (127.19.246.194:40887): tablet bb420e3f81004059ab16afe998c885d8 (table test-table [id=4cc1713cd33c4b7d9c203431aabde0e2]) successfully deleted
I20260810 03:20:21.874857 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 31887
I20260810 03:20:21.900544 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 32040
Aug 10 03:20:21 dist-test-slave-btw8 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:20:21.927525 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:21.901898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.19.246.194:40887)
0810 03:20:21.902749 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:21.902756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:21.902778 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:20:21.902879 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:21.902885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:21.903004 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:21.903153 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:21.906001 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.907443 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.910279 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:21.911021 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:21.911193 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:21.917575 (+ 6382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:21.917598 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:21.917617 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:21.917685 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:21.922616 (+ 4931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:21.924238 (+ 1622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:21.924249 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:21.924252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:21.924521 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:21.925123 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:21.925127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:21.925130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:21.925747 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:21.926164 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:21.926654 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 03:20:21.926984 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643}
I20260810 03:20:21.928953 32177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:22.070523 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 32040
I20260810 03:20:22.096513 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 32189
Aug 10 03:20:22 dist-test-slave-btw8 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:20:22.119896 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:20:22.097555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.19.246.195:43037)
0810 03:20:22.098194 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0810 03:20:22.098199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:20:22.098220 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:20:22.098286 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:20:22.098290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:20:22.098367 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:20:22.098490 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:20:22.100311 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:22.101173 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:22.103105 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:20:22.104009 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:20:22.104313 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:20:22.110243 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:20:22.110264 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:20:22.110283 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:20:22.110337 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:20:22.114077 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:22.115877 (+ 1800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:22.115900 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:22.115905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:22.116358 (+ 453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:20:22.117275 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:20:22.117281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:20:22.117285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:20:22.117936 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:20:22.118273 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 03:20:22.119074 (+ 801us) server_negotiation.cc:299] Negotiation successful
0810 03:20:22.119582 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":400,"threads_started":1}
I20260810 03:20:22.120759 32330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:22.220198 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 32189
I20260810 03:20:22.241806 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 31792
I20260810 03:20:22.243659 31856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:20:22.372350 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 31792
2026-08-10T03:20:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5947 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:20:22 dist-test-slave-btw8 krb5kdc[32394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:20:22 dist-test-slave-btw8 krb5kdc[32394](info): set up 2 sockets
Aug 10 03:20:22 dist-test-slave-btw8 krb5kdc[32394](info): commencing operation
krb5kdc: starting...
W20260810 03:20:23.697849 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.001s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:20:23 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:20:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/rangeradmin_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/rangeradmin_127.19.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/rangerlookup_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/rangerlookup_127.19.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/HTTP_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/HTTP_127.19.246.212@KRBTEST.COM.keytab.
I20260810 03:20:24.233649 20443 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZHDeAK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/postgres -l logfile start
2026-08-10 03:20:27.963 UTC [32429] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 03:20:27.963 UTC [32429] LOG: listening on IPv4 address "127.19.246.212", port 51039
2026-08-10 03:20:27.968 UTC [32429] LOG: listening on Unix socket "/tmp/.s.PGSQL.51039"
2026-08-10 03:20:27.994 UTC [32431] LOG: database system was shut down at 2026-08-10 03:20:25 UTC
2026-08-10 03:20:27.999 UTC [32429] LOG: database system is ready to accept connections
I20260810 03:20:29.121443 20443 mini_postgres.cc:96] Postgres bound to 51039
2026-08-10 03:20:29.153 UTC [32440] FATAL: database "slave" does not exist
127.19.246.212:51039 - accepting connections
I20260810 03:20:29.154407 20443 mini_ranger.cc:161] Starting Ranger...
I20260810 03:20:29.194530 20443 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 03:20:29.858587 20443 mini_ranger.cc:87] Created ranger Postgres database
I20260810 03:20:29.858783 20443 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-admin
2026-08-10 03:20:30,257 [I] DB FLAVOR :POSTGRES
2026-08-10 03:20:30,258 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:20:30,258 [I] Checking connection..
2026-08-10 03:20:30,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 03:20:30,543 [I] Checking connection passed.
2026-08-10 03:20:30,543 [I] --------- Verifying version history table ---------
2026-08-10 03:20:30,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:20:30,850 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:20:30,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:20:31,160 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:20:31,161 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 03:20:31,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 03:20:31.455 UTC [32512] WARNING: there is no transaction in progress
2026-08-10 03:20:31,465 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 03:20:31,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:20:31,786 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 03:20:31,786 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:20:31,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 03:20:32,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 03:20:32,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-btw8.c.gcp-upstream.internal','N') ;"
2026-08-10 03:20:32,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:20:33,007 [I] Table x_portal_user does not exist in database ranger
2026-08-10 03:20:33,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:20:33,280 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 03:20:33,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:20:33,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:20:33,872 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 03:20:33,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 03:20:35.150 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.174 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.198 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.221 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.244 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.276 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.348 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.359 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.378 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.391 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.405 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.425 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.440 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.450 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.460 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:35.470 UTC [32636] WARNING: there is no transaction in progress
2026-08-10 03:20:36,437 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 03:20:36,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 03:20:36,746 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 03:20:36,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:20:37,069 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 03:20:37,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:20:37,366 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 03:20:37,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:20:37,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:20:37,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-btw8.c.gcp-upstream.internal';"
2026-08-10 03:20:38,296 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 03:20:38,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 03:20:38,599 [I] DB_PATCHES have already been applied
I20260810 03:20:38.606757 20443 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZHDeAK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 03:20:38.606949 20443 mini_ranger.cc:193] Using host: 127.19.246.212
I20260810 03:20:38.632570 20443 mini_ranger.cc:227] Ranger admin URL: http://127.19.246.212:53881
Aug 10, 2026 3:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:20:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 03:20:39.957762 20443 mini_ranger.cc:160] Time spent starting Ranger: real 10.803s user 0.009s sys 0.069s
Aug 10, 2026 3:21:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:21:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 3:21:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:21:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:21:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 03:21:07.169643 20443 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/rangerkms_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/rangerkms_127.19.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.246.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.246.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.246.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/HTTP_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.246.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/HTTP_127.19.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 03:21:07 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332067, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 03:21:07.415633 20443 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 03:21:07.449247 20443 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 03:21:08.683009 20443 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 03:21:08.683169 20443 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms
I20260810 03:21:08.683396 20443 mini_ranger_kms.cc:227] Using postgres at 127.19.246.212:51039
2026-08-10 03:21:09,293 [I] DB FLAVOR :POSTGRES
2026-08-10 03:21:09,293 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:21:09,293 [I] Checking connection
2026-08-10 03:21:09,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 03:21:09,572 [I] connection success
2026-08-10 03:21:09,572 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 03:21:09,572 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 03:21:09,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 03:21:09,861 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 03:21:09,862 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:21:09,862 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 03:21:09,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:51039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 03:21:10,212 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 03:21:10.811193 20443 mini_ranger_kms.cc:310] Created kms service
I20260810 03:21:11.045737 20443 mini_ranger_kms.cc:326] Created kudu user
I20260810 03:21:11.099691 20443 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 03:21:11 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332067, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:21:11.637064 20443 mini_ranger_kms.cc:384] Added ranger policy
I20260810 03:21:11.669528 20443 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 03:21:11.669708 20443 mini_ranger_kms.cc:240] Using host: 127.19.246.212
I20260810 03:21:11.703058 20443 mini_ranger_kms.cc:276] Ranger KMS PID: 364
I20260810 03:21:11.703330 20443 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.246.212:60981
Aug 10, 2026 3:21:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:21:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:21:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:21:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 03:21:14.327441 20443 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.912s user 0.141s sys 0.514s
I20260810 03:21:14.327826 20443 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 03:21:14.328087 20443 mini_ranger_kms.cc:401] 127.19.246.212:60981/kms/v1/keys
Aug 10 03:21:14 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332074, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.246.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 03:21:14 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332074, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.246.212@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
Aug 10 03:21:15 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332074, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.246.212@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
Aug 10 03:21:17 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:21:18.081951 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:38401
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:38401
--ranger_config_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:60981/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:21:18.671998 415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:21:18.672361 415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:21:18.672524 415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:21:18.682933 415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:21:18.683250 415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:21:18.683426 415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:21:18.683581 415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:21:18.683684 415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:21:18.700155 415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.246.212:60981/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:38401
--ranger_config_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:38401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:21:18.701910 415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:21:18.703764 415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:21:18.725189 420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:21:18.725458 421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:21:18.725508 423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:21:18.727552 415 server_base.cc:1034] running on GCE node
I20260810 03:21:18.728713 415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:21:18.730765 415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:21:18.732126 415 hybrid_clock.cc:648] HybridClock initialized: now 1786332078732019 us; error 65 us; skew 500 ppm
I20260810 03:21:18.732968 415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:21:18 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332078, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:21:18.739207 415 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:21:18.742097 415 webserver.cc:460] Webserver started at http://127.19.246.254:35877/ using document root <none> and password file <none>
I20260810 03:21:18.742978 415 fs_manager.cc:359] Metadata directory not provided
I20260810 03:21:18.743257 415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:21:18.743891 415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:21:18 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332078, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:21:18.837779 415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "2f150338cdd345a9a7e94bfc98aa99f2"
format_stamp: "Formatted at 2026-08-10 03:21:18 on dist-test-slave-btw8"
server_key: "d12e9295e087ff139e3e38515f1e00a7"
server_key_iv: "79b8190b63f453f530b5a8d8fcf834f9"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:18.839947 415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "2f150338cdd345a9a7e94bfc98aa99f2"
format_stamp: "Formatted at 2026-08-10 03:21:18 on dist-test-slave-btw8"
server_key: "d12e9295e087ff139e3e38515f1e00a7"
server_key_iv: "79b8190b63f453f530b5a8d8fcf834f9"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:18.848963 415 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260810 03:21:18.877467 431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:18.879480 415 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.003s
I20260810 03:21:18.879784 415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "2f150338cdd345a9a7e94bfc98aa99f2"
format_stamp: "Formatted at 2026-08-10 03:21:18 on dist-test-slave-btw8"
server_key: "d12e9295e087ff139e3e38515f1e00a7"
server_key_iv: "79b8190b63f453f530b5a8d8fcf834f9"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:18.880110 415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:21:18.921669 415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:21:18.930658 415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:21:18.931330 415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:21:18.980170 415 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:38401
I20260810 03:21:18.980208 488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:38401 every 8 connection(s)
I20260810 03:21:18.982406 415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:21:18.983415 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 415
I20260810 03:21:18.983732 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:21:18.985092 20443 external_mini_cluster.cc:1428] Setting key fb04b8bfcaadd539b414127b75342a8d
I20260810 03:21:18.990638 489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:21:19 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:21:19.009703 489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2: Bootstrap starting.
I20260810 03:21:19.014920 489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2: Neither blocks nor log segments found. Creating new log.
I20260810 03:21:19.017733 489 log.cc:824] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:21:19.021240 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:18.988115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.19.246.254:38401)
0810 03:21:18.989839 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:18.989860 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:18.990278 (+ 418us) server_negotiation.cc:407] Connection header received
0810 03:21:18.993212 (+ 2934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:18.993238 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:18.993711 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:18.994182 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:21:18.996172 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:18.998479 (+ 2307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:19.001297 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:19.002422 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:19.002629 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:19.009071 (+ 6442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:21:19.009082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:21:19.009102 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:21:19.009170 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:21:19.013690 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:19.015264 (+ 1574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:19.015273 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:19.015281 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:19.015658 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:19.016524 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:19.016528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:19.016530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:19.017396 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:21:19.017744 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:19.018512 (+ 768us) server_negotiation.cc:299] Negotiation successful
0810 03:21:19.019156 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1386,"thread_start_us":858,"threads_started":1}
I20260810 03:21:19.023289 489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2: No bootstrap required, opened a new log
I20260810 03:21:19.035939 489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f150338cdd345a9a7e94bfc98aa99f2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38401 } }
I20260810 03:21:19.036459 489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:21:19.036594 489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f150338cdd345a9a7e94bfc98aa99f2, State: Initialized, Role: FOLLOWER
I20260810 03:21:19.037350 489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f150338cdd345a9a7e94bfc98aa99f2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38401 } }
I20260810 03:21:19.037657 489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:21:19.037866 489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:21:19.038106 489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:21:19.041195 489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f150338cdd345a9a7e94bfc98aa99f2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38401 } }
I20260810 03:21:19.042562 489 leader_election.cc:302] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f150338cdd345a9a7e94bfc98aa99f2; no voters:
I20260810 03:21:19.044494 489 leader_election.cc:288] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:21:19.044744 495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:21:19.047569 495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2f150338cdd345a9a7e94bfc98aa99f2, State: Running, Role: LEADER
I20260810 03:21:19.048650 495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f150338cdd345a9a7e94bfc98aa99f2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38401 } }
I20260810 03:21:19.049047 489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:21:19.053381 497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f150338cdd345a9a7e94bfc98aa99f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f150338cdd345a9a7e94bfc98aa99f2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38401 } } }
I20260810 03:21:19.053361 496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f150338cdd345a9a7e94bfc98aa99f2. Latest consensus state: current_term: 1 leader_uuid: "2f150338cdd345a9a7e94bfc98aa99f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f150338cdd345a9a7e94bfc98aa99f2" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38401 } } }
I20260810 03:21:19.054481 497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:21:19.054487 496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2 [sys.catalog]: This master's current role is: LEADER
I20260810 03:21:19.062546 500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:21:19.073791 500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:21:19.074940 489 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:21:19.117761 500 catalog_manager.cc:1347] Generated new cluster ID: 4b00bd9687dd4684ae3e8a3f5d0c0cb8
I20260810 03:21:19.118068 500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:21:19.137126 500 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:21:19.139271 500 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:21:19.152652 500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2: Generated new TSK 0
I20260810 03:21:19.154196 500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:21:20.188855 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:60981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38401
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:21:20 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332080, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:21:20.870481 537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:21:20.871659 537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:21:20.872041 537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:21:20.886934 537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:21:20.887230 537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:21:20.887633 537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:21:20.911852 537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.246.212:60981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:21:20.913864 537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:21:20.915763 537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:21:20.950584 548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:21:20.956297 547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:21:20.957233 550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:21:20.958534 537 server_base.cc:1034] running on GCE node
I20260810 03:21:20.959789 537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:21:20.961373 537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:21:20.962818 537 hybrid_clock.cc:648] HybridClock initialized: now 1786332080962688 us; error 82 us; skew 500 ppm
I20260810 03:21:20.963307 537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:21:20 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332080, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:21:20.968674 537 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:21:20.971849 537 webserver.cc:460] Webserver started at http://127.19.246.193:41253/ using document root <none> and password file <none>
I20260810 03:21:20.972828 537 fs_manager.cc:359] Metadata directory not provided
I20260810 03:21:20.973102 537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:21:20.973888 537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:21:20 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332080, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:21:21.003698 537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "20cae58e1a9148b3abc7bcec6c637f4d"
format_stamp: "Formatted at 2026-08-10 03:21:20 on dist-test-slave-btw8"
server_key: "d0409aae2b4be53d426f22b507f99830"
server_key_iv: "e7e40301472b3d0b39f499b0c254c0de"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:21.006160 537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "20cae58e1a9148b3abc7bcec6c637f4d"
format_stamp: "Formatted at 2026-08-10 03:21:20 on dist-test-slave-btw8"
server_key: "d0409aae2b4be53d426f22b507f99830"
server_key_iv: "e7e40301472b3d0b39f499b0c254c0de"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:21.019500 537 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.009s sys 0.005s
I20260810 03:21:21.081327 557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:21.084843 537 fs_manager.cc:725] Time spent opening block manager: real 0.038s user 0.022s sys 0.005s
I20260810 03:21:21.085402 537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "20cae58e1a9148b3abc7bcec6c637f4d"
format_stamp: "Formatted at 2026-08-10 03:21:20 on dist-test-slave-btw8"
server_key: "d0409aae2b4be53d426f22b507f99830"
server_key_iv: "e7e40301472b3d0b39f499b0c254c0de"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:21.086197 537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:21:21.116508 537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:21:21.126889 537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:21:21.128684 537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:21:21.131089 537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:21:21.141377 537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:21:21.141561 537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:21.141927 537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:21:21.142060 537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:21:21 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332080, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:21:21.566918 537 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:39627
I20260810 03:21:21.566998 683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:39627 every 8 connection(s)
I20260810 03:21:21.569247 537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:21:21.578406 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 537
I20260810 03:21:21.578683 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:21:21.579726 20443 external_mini_cluster.cc:1428] Setting key fa6ab0840161cf176845089f2dd3b21a
Aug 10 03:21:21 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332080, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
I20260810 03:21:21.647109 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:21.576251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:38401 (local address 127.19.246.193:43857)
0810 03:21:21.582271 (+ 6020us) negotiation.cc:107] Waiting for socket to connect
0810 03:21:21.582346 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 03:21:21.604704 (+ 22358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:21:21.606358 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:21:21.606387 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:21:21.608624 (+ 2237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:21:21.610703 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:21.610756 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:21.613409 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:21.613418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:21.618079 (+ 4661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:21.618170 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:21.620464 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:21.620477 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:21.621026 (+ 549us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:21.622627 (+ 1601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:21:21.622670 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:21:21.629958 (+ 7288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:21:21.638384 (+ 8426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:21:21.638404 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:21:21.638435 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:21:21.640187 (+ 1752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:21:21.641074 (+ 887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:21:21.641082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:21:21.641086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:21:21.641481 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:21:21.642618 (+ 1137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:21:21.642630 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:21:21.643402 (+ 772us) client_negotiation.cc:769] Sending connection context
0810 03:21:21.643889 (+ 487us) client_negotiation.cc:240] Negotiation successful
0810 03:21:21.644787 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"thread_start_us":856,"threads_started":1}
I20260810 03:21:21.649721 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:21.574706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:43857 (local address 127.19.246.254:38401)
0810 03:21:21.575788 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:21.575793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:21.589188 (+ 13395us) server_negotiation.cc:407] Connection header received
0810 03:21:21.605296 (+ 16108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:21.605304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:21.605425 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:21.605545 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:21:21.611117 (+ 5572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:21.612851 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:21.618491 (+ 5640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:21.619624 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:21.619787 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:21.630558 (+ 10771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:21:21.630567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:21:21.630572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:21:21.630630 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:21:21.637887 (+ 7257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:21.640472 (+ 2585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:21.640481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:21.640485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:21.640767 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:21.641670 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:21.641677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:21.641681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:21.642255 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:21:21.642478 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:21.648621 (+ 6143us) server_negotiation.cc:299] Negotiation successful
0810 03:21:21.649138 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":671,"threads_started":1}
I20260810 03:21:21.652061 684 heartbeater.cc:347] Connected to a master server at 127.19.246.254:38401
I20260810 03:21:21.652535 684 heartbeater.cc:464] Registering TS with master...
I20260810 03:21:21.653692 684 heartbeater.cc:511] Master 127.19.246.254:38401 requested a full tablet report, sending...
I20260810 03:21:21.658737 454 ts_manager.cc:194] Registered new tserver with Master: 20cae58e1a9148b3abc7bcec6c637f4d (127.19.246.193:39627)
I20260810 03:21:21.661791 454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:43857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
W20260810 03:21:21.696233 20443 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.19.246.194: real 0.112s user 0.003s sys 0.028s
I20260810 03:21:21.697525 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:60981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38401
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:21:21 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332080, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:21:22.667870 684 heartbeater.cc:503] Master 127.19.246.254:38401 was elected leader, sending a full tablet report...
W20260810 03:21:22.682291 691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:21:22.682878 691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:21:22.683194 691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:21:22.696640 691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:21:22.696967 691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:21:22.697328 691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:21:22.713583 691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.246.212:60981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:21:22.715472 691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:21:22.717228 691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:21:22.740716 706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:21:22.743157 707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:21:22.743625 709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:21:22.746693 691 server_base.cc:1034] running on GCE node
I20260810 03:21:22.748436 691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:21:22.750459 691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:21:22.751984 691 hybrid_clock.cc:648] HybridClock initialized: now 1786332082751847 us; error 111 us; skew 500 ppm
I20260810 03:21:22.752547 691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:21:22 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332082, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:21:22.766357 691 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:21:22.769217 691 webserver.cc:460] Webserver started at http://127.19.246.194:43633/ using document root <none> and password file <none>
I20260810 03:21:22.770324 691 fs_manager.cc:359] Metadata directory not provided
I20260810 03:21:22.770869 691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:21:22.771818 691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:21:22 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332082, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:21:22.793797 691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "6c0dca9320854598b2307e1bb438f94b"
format_stamp: "Formatted at 2026-08-10 03:21:22 on dist-test-slave-btw8"
server_key: "502055f2c4611aab1993b7012beb863c"
server_key_iv: "f6c9be1c6e485af10979c58145c6fe1f"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:22.795198 691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "6c0dca9320854598b2307e1bb438f94b"
format_stamp: "Formatted at 2026-08-10 03:21:22 on dist-test-slave-btw8"
server_key: "502055f2c4611aab1993b7012beb863c"
server_key_iv: "f6c9be1c6e485af10979c58145c6fe1f"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:22.804975 691 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.002s
I20260810 03:21:22.826484 717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:22.829085 691 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:21:22.829391 691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "6c0dca9320854598b2307e1bb438f94b"
format_stamp: "Formatted at 2026-08-10 03:21:22 on dist-test-slave-btw8"
server_key: "502055f2c4611aab1993b7012beb863c"
server_key_iv: "f6c9be1c6e485af10979c58145c6fe1f"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:22.829689 691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:21:22.856997 691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:21:22.864015 691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:21:22.864605 691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:21:22.866587 691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:21:22.869418 691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:21:22.869623 691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:22.869935 691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:21:22.870098 691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:22.990653 691 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:45281
I20260810 03:21:22.990705 839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:45281 every 8 connection(s)
I20260810 03:21:22.993183 691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:21:22.998598 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 691
I20260810 03:21:22.998839 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:21:22.999866 20443 external_mini_cluster.cc:1428] Setting key 7a0a7fd8ee4b308133b99d2b01c1ac16
Aug 10 03:21:23 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332082, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:21:23.032513 843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:22.999815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:56271 (local address 127.19.246.254:38401)
0810 03:21:23.002055 (+ 2240us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:23.002062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:23.002085 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:21:23.005621 (+ 3536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:23.005631 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:23.005750 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:23.005893 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:21:23.008900 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:23.010753 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:23.013669 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:23.014915 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:23.015107 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:23.022266 (+ 7159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:21:23.022271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:21:23.022275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:21:23.022336 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:21:23.027113 (+ 4777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:23.028493 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:23.028498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:23.028501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:23.028794 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:23.029480 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:23.029485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:23.029488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:23.029977 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:21:23.030212 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:23.031308 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0810 03:21:23.031695 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2107,"thread_start_us":2879,"threads_started":1}
I20260810 03:21:23.033763 842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:22.999254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:38401 (local address 127.19.246.194:56271)
0810 03:21:23.001613 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0810 03:21:23.001671 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:21:23.005182 (+ 3511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:21:23.006324 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:21:23.006335 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:21:23.007236 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:21:23.008585 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:23.008613 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:23.011044 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:23.011049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:23.013360 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:23.013417 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:23.015771 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:23.015785 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:23.016162 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:23.017462 (+ 1300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:21:23.017502 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:21:23.021884 (+ 4382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:21:23.027343 (+ 5459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:21:23.027376 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:21:23.027399 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:21:23.028248 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:21:23.028945 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:21:23.028951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:21:23.028954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:21:23.029324 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:21:23.030208 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:21:23.030219 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:21:23.030820 (+ 601us) client_negotiation.cc:769] Sending connection context
0810 03:21:23.031171 (+ 351us) client_negotiation.cc:240] Negotiation successful
0810 03:21:23.032109 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"thread_start_us":1412,"threads_started":1}
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
I20260810 03:21:23.037220 840 heartbeater.cc:347] Connected to a master server at 127.19.246.254:38401
I20260810 03:21:23.037755 840 heartbeater.cc:464] Registering TS with master...
I20260810 03:21:23.039079 840 heartbeater.cc:511] Master 127.19.246.254:38401 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
I20260810 03:21:23.041409 454 ts_manager.cc:194] Registered new tserver with Master: 6c0dca9320854598b2307e1bb438f94b (127.19.246.194:45281)
I20260810 03:21:23.042977 454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:56271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:21:23.084887 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:60981/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38401
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:21:23.877765 846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:21:23.878170 846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:21:23.878342 846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:21:23.893061 846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:21:23.893317 846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:21:23.893508 846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:21:23.914312 846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.246.212:60981/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:38401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:21:23.916121 846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:21:23.917766 846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:21:23.943216 851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:21:23.943315 852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:21:23.943444 854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:21:23.947270 846 server_base.cc:1034] running on GCE node
I20260810 03:21:23.948311 846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:21:23.950091 846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:21:23.951566 846 hybrid_clock.cc:648] HybridClock initialized: now 1786332083951402 us; error 71 us; skew 500 ppm
I20260810 03:21:23.952308 846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:21:23 dist-test-slave-btw8 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332083, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:21:23.958658 846 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:21:23.961226 846 webserver.cc:460] Webserver started at http://127.19.246.195:34245/ using document root <none> and password file <none>
I20260810 03:21:23.962371 846 fs_manager.cc:359] Metadata directory not provided
I20260810 03:21:23.962641 846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:21:23.963097 846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:21:23 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332083, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:21:23.990854 846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "ecf54dcf0edc4aa98797a485574c20cf"
format_stamp: "Formatted at 2026-08-10 03:21:23 on dist-test-slave-btw8"
server_key: "dc969e7e1b2bd892f3e0ed17bb4b4444"
server_key_iv: "88cb2f50754e206269d42d44b112e525"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:23.992125 846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "ecf54dcf0edc4aa98797a485574c20cf"
format_stamp: "Formatted at 2026-08-10 03:21:23 on dist-test-slave-btw8"
server_key: "dc969e7e1b2bd892f3e0ed17bb4b4444"
server_key_iv: "88cb2f50754e206269d42d44b112e525"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:24.001995 846 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s
I20260810 03:21:24.026244 861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:24.028838 846 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:21:24.029206 846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "ecf54dcf0edc4aa98797a485574c20cf"
format_stamp: "Formatted at 2026-08-10 03:21:23 on dist-test-slave-btw8"
server_key: "dc969e7e1b2bd892f3e0ed17bb4b4444"
server_key_iv: "88cb2f50754e206269d42d44b112e525"
server_key_version: "kuduclusterkey@0"
I20260810 03:21:24.029626 846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:21:24.047745 840 heartbeater.cc:503] Master 127.19.246.254:38401 was elected leader, sending a full tablet report...
I20260810 03:21:24.059442 846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:21:24.067844 846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:21:24.068997 846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:21:24.071842 846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:21:24.075928 846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:21:24.076146 846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:24.076422 846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:21:24.076592 846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:21:24.195092 846 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:43785
I20260810 03:21:24.195159 981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:43785 every 8 connection(s)
I20260810 03:21:24.197398 846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:21:24.207301 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 846
I20260810 03:21:24.207602 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:21:24.208489 20443 external_mini_cluster.cc:1428] Setting key f6bcb4543101f2b8d9cac73d91616e6e
Aug 10 03:21:24 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332083, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:21:24.240036 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.202412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:59217 (local address 127.19.246.254:38401)
0810 03:21:24.203511 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.203516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.206278 (+ 2762us) server_negotiation.cc:407] Connection header received
0810 03:21:24.212108 (+ 5830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.212115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.212362 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.212603 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:21:24.216188 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.217917 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.221698 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.222317 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.222514 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.228693 (+ 6179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:21:24.228702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:21:24.228707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:21:24.228769 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:21:24.234329 (+ 5560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:24.236066 (+ 1737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:24.236071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:24.236074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:24.236401 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:24.237202 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:24.237207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:24.237216 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:24.237774 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:21:24.237920 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.239052 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.239306 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":702,"threads_started":1}
I20260810 03:21:24.241534 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.203561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:38401 (local address 127.19.246.195:59217)
0810 03:21:24.205863 (+ 2302us) negotiation.cc:107] Waiting for socket to connect
0810 03:21:24.205964 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0810 03:21:24.211580 (+ 5616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:21:24.213252 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:21:24.213266 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:21:24.214413 (+ 1147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:21:24.215776 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.215807 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.218374 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.218382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.221415 (+ 3033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.221446 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.222802 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.222811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.223181 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.224232 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:21:24.224266 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:21:24.228389 (+ 4123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:21:24.234613 (+ 6224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:21:24.234624 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:21:24.234650 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:21:24.235661 (+ 1011us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:21:24.236673 (+ 1012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:21:24.236679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:21:24.236683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:21:24.236971 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:21:24.237964 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:21:24.237971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:21:24.238576 (+ 605us) client_negotiation.cc:769] Sending connection context
0810 03:21:24.238897 (+ 321us) client_negotiation.cc:240] Negotiation successful
0810 03:21:24.239505 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1654,"thread_start_us":1146,"threads_started":1}
I20260810 03:21:24.243935 982 heartbeater.cc:347] Connected to a master server at 127.19.246.254:38401
I20260810 03:21:24.244407 982 heartbeater.cc:464] Registering TS with master...
I20260810 03:21:24.246208 982 heartbeater.cc:511] Master 127.19.246.254:38401 requested a full tablet report, sending...
I20260810 03:21:24.248880 454 ts_manager.cc:194] Registered new tserver with Master: ecf54dcf0edc4aa98797a485574c20cf (127.19.246.195:43785)
I20260810 03:21:24.250526 454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:59217
I20260810 03:21:24.260757 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:21:24.299219 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.270918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.19.246.254:38401)
0810 03:21:24.271488 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.271498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.273054 (+ 1556us) server_negotiation.cc:407] Connection header received
0810 03:21:24.273155 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.273159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.273512 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.273637 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:21:24.277093 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.278303 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.281459 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.282368 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.282857 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.288525 (+ 5668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:21:24.288533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:21:24.288536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:21:24.288604 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:21:24.293945 (+ 5341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:24.295543 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:24.295548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:24.295551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:24.295962 (+ 411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:24.296767 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:24.296775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:24.296778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:24.297418 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:21:24.297795 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.298415 (+ 620us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.298718 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260810 03:21:24.307051 454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:21:24.322214 454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:21:24.373217 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.350978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.19.246.195:43785)
0810 03:21:24.355599 (+ 4621us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.355615 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.355712 (+ 97us) server_negotiation.cc:407] Connection header received
0810 03:21:24.355868 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.355884 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.356407 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.357694 (+ 1287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:21:24.359582 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.362948 (+ 3366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.368660 (+ 5712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.370920 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.371182 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.372204 (+ 1022us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.372285 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.372482 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1933,"thread_start_us":1509,"threads_started":1}
I20260810 03:21:24.374064 996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.349837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:43785 (local address 127.0.0.1:58254)
0810 03:21:24.354669 (+ 4832us) negotiation.cc:107] Waiting for socket to connect
0810 03:21:24.354740 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 03:21:24.355081 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:21:24.356758 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:21:24.356767 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:21:24.356807 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:21:24.358898 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.358942 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.363234 (+ 4292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.363245 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.368321 (+ 5076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.368343 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.371300 (+ 2957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.371309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.371667 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.371742 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 03:21:24.371890 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 03:21:24.373021 (+ 1131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4350,"thread_start_us":3935,"threads_started":1}
I20260810 03:21:24.381071 912 tablet_service.cc:1467] Processing CreateTablet for tablet 82ac8931371a43d2bffa32f767bf91b3 (DEFAULT_TABLE table=test-table [id=68ffec01160c421b9b26daff76bc114b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:21:24.382509 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.352525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.19.246.193:39627)
0810 03:21:24.358681 (+ 6156us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.358691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.359593 (+ 902us) server_negotiation.cc:407] Connection header received
0810 03:21:24.362254 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.362271 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.362796 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.363507 (+ 711us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:21:24.369919 (+ 6412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.372876 (+ 2957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.377380 (+ 4504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.380313 (+ 2933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.380753 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.381074 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.381366 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.381669 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5831,"thread_start_us":5387,"threads_started":1}
I20260810 03:21:24.382793 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.353907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:39627 (local address 127.0.0.1:36536)
0810 03:21:24.359101 (+ 5194us) negotiation.cc:107] Waiting for socket to connect
0810 03:21:24.359126 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:21:24.359326 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:21:24.363273 (+ 3947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:21:24.363282 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:21:24.363329 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:21:24.369444 (+ 6115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.369464 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.373134 (+ 3670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.373152 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.377087 (+ 3935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.377104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.380770 (+ 3666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.380778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.381047 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.381107 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 03:21:24.381226 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:21:24.381429 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4984,"mutex_wait_us":7,"thread_start_us":3502,"threads_started":1}
I20260810 03:21:24.384436 912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ac8931371a43d2bffa32f767bf91b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:21:24.389587 612 tablet_service.cc:1467] Processing CreateTablet for tablet 82ac8931371a43d2bffa32f767bf91b3 (DEFAULT_TABLE table=test-table [id=68ffec01160c421b9b26daff76bc114b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:21:24.391661 612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ac8931371a43d2bffa32f767bf91b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:21:24.401381 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.359104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:45281 (local address 127.0.0.1:36724)
0810 03:21:24.369923 (+ 10819us) negotiation.cc:107] Waiting for socket to connect
0810 03:21:24.369994 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 03:21:24.370199 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:21:24.371892 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:21:24.371898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:21:24.371933 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:21:24.375083 (+ 3150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.375113 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.381661 (+ 6548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.381668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.395818 (+ 14150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.395846 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.399824 (+ 3978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.399834 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.400211 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.400254 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 03:21:24.400386 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 03:21:24.400604 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10589,"thread_start_us":10197,"threads_started":1}
I20260810 03:21:24.401588 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.359593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.19.246.194:45281)
0810 03:21:24.365908 (+ 6315us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.365921 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.370161 (+ 4240us) server_negotiation.cc:407] Connection header received
0810 03:21:24.370503 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.370518 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.370954 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.371410 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:21:24.375678 (+ 4268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.381262 (+ 5584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.396099 (+ 14837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.398888 (+ 2789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.399436 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.399833 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.400625 (+ 792us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.400937 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5956,"thread_start_us":5441,"threads_started":1}
I20260810 03:21:24.407721 1003 tablet_bootstrap.cc:492] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Bootstrap starting.
I20260810 03:21:24.410794 770 tablet_service.cc:1467] Processing CreateTablet for tablet 82ac8931371a43d2bffa32f767bf91b3 (DEFAULT_TABLE table=test-table [id=68ffec01160c421b9b26daff76bc114b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:21:24.413009 770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ac8931371a43d2bffa32f767bf91b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:21:24.415763 1003 tablet_bootstrap.cc:654] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Neither blocks nor log segments found. Creating new log.
I20260810 03:21:24.418529 1003 log.cc:824] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Log is configured to *not* fsync() on all Append() calls
I20260810 03:21:24.418602 1004 tablet_bootstrap.cc:492] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Bootstrap starting.
I20260810 03:21:24.424568 1003 tablet_bootstrap.cc:492] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: No bootstrap required, opened a new log
I20260810 03:21:24.425122 1003 ts_tablet_manager.cc:1397] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260810 03:21:24.427255 1004 tablet_bootstrap.cc:654] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Neither blocks nor log segments found. Creating new log.
I20260810 03:21:24.430825 1004 log.cc:824] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:21:24.438308 1004 tablet_bootstrap.cc:492] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: No bootstrap required, opened a new log
I20260810 03:21:24.439129 1004 ts_tablet_manager.cc:1397] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.012s
I20260810 03:21:24.440771 1007 tablet_bootstrap.cc:492] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Bootstrap starting.
I20260810 03:21:24.442315 1003 raft_consensus.cc:348] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.443008 1003 raft_consensus.cc:374] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:21:24.443387 1003 raft_consensus.cc:729] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf54dcf0edc4aa98797a485574c20cf, State: Initialized, Role: FOLLOWER
I20260810 03:21:24.444566 1003 consensus_queue.cc:260] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.448468 982 heartbeater.cc:503] Master 127.19.246.254:38401 was elected leader, sending a full tablet report...
I20260810 03:21:24.451560 1007 tablet_bootstrap.cc:654] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Neither blocks nor log segments found. Creating new log.
I20260810 03:21:24.452104 1003 ts_tablet_manager.cc:1428] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Time spent starting tablet: real 0.027s user 0.020s sys 0.005s
I20260810 03:21:24.455511 1007 log.cc:824] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:21:24.461684 1007 tablet_bootstrap.cc:492] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: No bootstrap required, opened a new log
I20260810 03:21:24.461594 1004 raft_consensus.cc:348] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.462239 1007 ts_tablet_manager.cc:1397] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.009s
I20260810 03:21:24.462488 1004 raft_consensus.cc:374] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:21:24.462782 1004 raft_consensus.cc:729] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cae58e1a9148b3abc7bcec6c637f4d, State: Initialized, Role: FOLLOWER
I20260810 03:21:24.463912 1004 consensus_queue.cc:260] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.468569 1004 ts_tablet_manager.cc:1428] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Time spent starting tablet: real 0.029s user 0.017s sys 0.011s
I20260810 03:21:24.478729 1007 raft_consensus.cc:348] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.479410 1007 raft_consensus.cc:374] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:21:24.479633 1007 raft_consensus.cc:729] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0dca9320854598b2307e1bb438f94b, State: Initialized, Role: FOLLOWER
I20260810 03:21:24.480409 1007 consensus_queue.cc:260] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.483481 1007 ts_tablet_manager.cc:1428] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Time spent starting tablet: real 0.021s user 0.012s sys 0.012s
W20260810 03:21:24.498852 841 tablet.cc:2369] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:21:24.576788 685 tablet.cc:2369] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:21:24.642980 1008 raft_consensus.cc:482] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:21:24.643663 1008 raft_consensus.cc:504] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.646170 1008 leader_election.cc:288] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20cae58e1a9148b3abc7bcec6c637f4d (127.19.246.193:39627), 6c0dca9320854598b2307e1bb438f94b (127.19.246.194:45281)
I20260810 03:21:24.657624 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.646653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:59669 (local address 127.19.246.193:39627)
0810 03:21:24.646884 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.646888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.646911 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:21:24.647326 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.647330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.647469 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.647601 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:21:24.648349 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.650352 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.654022 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.656139 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.656446 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.656551 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.657227 (+ 676us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.657344 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:21:24.657806 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.646559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:39627 (local address 127.19.246.195:59669)
0810 03:21:24.646813 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 03:21:24.646838 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:21:24.646980 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:21:24.647622 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:21:24.647627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:21:24.647658 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:21:24.647961 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.647984 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.650619 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.650630 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.653684 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.653701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.656407 (+ 2706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.656412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.656948 (+ 536us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.657053 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 03:21:24.657180 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 03:21:24.657371 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 03:21:24.660238 633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ac8931371a43d2bffa32f767bf91b3" candidate_uuid: "ecf54dcf0edc4aa98797a485574c20cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" is_pre_election: true
I20260810 03:21:24.660915 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.647877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:57041 (local address 127.19.246.194:45281)
0810 03:21:24.648257 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.648264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.648924 (+ 660us) server_negotiation.cc:407] Connection header received
0810 03:21:24.649104 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.649108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.649170 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.649299 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:21:24.650973 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.652726 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.656775 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.659164 (+ 2389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.659606 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.659741 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.660244 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.660443 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260810 03:21:24.660929 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.647429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:45281 (local address 127.19.246.195:57041)
0810 03:21:24.648770 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0810 03:21:24.648792 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:21:24.648951 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:21:24.649406 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:21:24.649412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:21:24.649441 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:21:24.650641 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.650656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.653020 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.653025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.656420 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:21:24.656440 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.659628 (+ 3188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.659636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:21:24.659919 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.659963 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:21:24.660111 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 03:21:24.660274 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":775,"threads_started":1}
I20260810 03:21:24.661325 633 raft_consensus.cc:2454] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf54dcf0edc4aa98797a485574c20cf in term 0.
I20260810 03:21:24.662850 865 leader_election.cc:302] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cae58e1a9148b3abc7bcec6c637f4d, ecf54dcf0edc4aa98797a485574c20cf; no voters:
I20260810 03:21:24.663511 1008 raft_consensus.cc:2781] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:21:24.663196 791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ac8931371a43d2bffa32f767bf91b3" candidate_uuid: "ecf54dcf0edc4aa98797a485574c20cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0dca9320854598b2307e1bb438f94b" is_pre_election: true
I20260810 03:21:24.663870 1008 raft_consensus.cc:482] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:21:24.664070 1008 raft_consensus.cc:3037] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:21:24.664582 791 raft_consensus.cc:2454] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf54dcf0edc4aa98797a485574c20cf in term 0.
I20260810 03:21:24.670162 1008 raft_consensus.cc:504] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.671295 1008 leader_election.cc:288] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [CANDIDATE]: Term 1 election: Requested vote from peers 20cae58e1a9148b3abc7bcec6c637f4d (127.19.246.193:39627), 6c0dca9320854598b2307e1bb438f94b (127.19.246.194:45281)
I20260810 03:21:24.672387 791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ac8931371a43d2bffa32f767bf91b3" candidate_uuid: "ecf54dcf0edc4aa98797a485574c20cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0dca9320854598b2307e1bb438f94b"
I20260810 03:21:24.672518 633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ac8931371a43d2bffa32f767bf91b3" candidate_uuid: "ecf54dcf0edc4aa98797a485574c20cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cae58e1a9148b3abc7bcec6c637f4d"
I20260810 03:21:24.672791 791 raft_consensus.cc:3037] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:21:24.672935 633 raft_consensus.cc:3037] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:21:24.678897 791 raft_consensus.cc:2454] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf54dcf0edc4aa98797a485574c20cf in term 1.
I20260810 03:21:24.678936 633 raft_consensus.cc:2454] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf54dcf0edc4aa98797a485574c20cf in term 1.
I20260810 03:21:24.679786 865 leader_election.cc:302] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0dca9320854598b2307e1bb438f94b, ecf54dcf0edc4aa98797a485574c20cf; no voters:
I20260810 03:21:24.680361 1008 raft_consensus.cc:2781] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:21:24.681334 1008 raft_consensus.cc:686] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 1 LEADER]: Becoming Leader. State: Replica: ecf54dcf0edc4aa98797a485574c20cf, State: Running, Role: LEADER
I20260810 03:21:24.682289 1008 consensus_queue.cc:237] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } }
I20260810 03:21:24.690279 453 catalog_manager.cc:5495] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf54dcf0edc4aa98797a485574c20cf (127.19.246.195). New cstate: current_term: 1 leader_uuid: "ecf54dcf0edc4aa98797a485574c20cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf54dcf0edc4aa98797a485574c20cf" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 } health_report { overall_health: UNKNOWN } } }
W20260810 03:21:24.701898 983 tablet.cc:2369] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:21:24.757177 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.745525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.19.246.195:43785)
0810 03:21:24.745986 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.745993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.746655 (+ 662us) server_negotiation.cc:407] Connection header received
0810 03:21:24.747005 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.747010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.747120 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.747280 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:21:24.748459 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.750208 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.753419 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.754165 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.754329 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.755461 (+ 1132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:21:24.756158 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:21:24.756306 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.756557 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.756653 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260810 03:21:24.772743 633 raft_consensus.cc:1261] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 1 FOLLOWER]: Refusing update from remote peer ecf54dcf0edc4aa98797a485574c20cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:21:24.772797 791 raft_consensus.cc:1261] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 1 FOLLOWER]: Refusing update from remote peer ecf54dcf0edc4aa98797a485574c20cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:21:24.774586 1013 consensus_queue.cc:1035] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0dca9320854598b2307e1bb438f94b" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 03:21:24.775799 1008 consensus_queue.cc:1035] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "20cae58e1a9148b3abc7bcec6c637f4d" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:21:24.823896 454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:21:24.824563 454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:21:24.832271 454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f150338cdd345a9a7e94bfc98aa99f2: Sending DeleteTablet for 3 replicas of tablet 82ac8931371a43d2bffa32f767bf91b3
I20260810 03:21:24.834627 612 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ac8931371a43d2bffa32f767bf91b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:21:24 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:36536
I20260810 03:21:24.834573 912 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ac8931371a43d2bffa32f767bf91b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:21:24 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:58254
I20260810 03:21:24.836310 770 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ac8931371a43d2bffa32f767bf91b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:21:24 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:36724
I20260810 03:21:24.845455 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 537
I20260810 03:21:24.846652 1028 tablet_replica.cc:330] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: stopping tablet replica
I20260810 03:21:24.846798 1031 tablet_replica.cc:330] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: stopping tablet replica
I20260810 03:21:24.847492 1031 raft_consensus.cc:2229] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:21:24.847729 1028 raft_consensus.cc:2229] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:21:24.848793 1031 raft_consensus.cc:2258] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:21:24.850287 1029 tablet_replica.cc:330] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: stopping tablet replica
I20260810 03:21:24.850157 1028 raft_consensus.cc:2258] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:21:24.851223 1029 raft_consensus.cc:2229] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:21:24.852465 1029 raft_consensus.cc:2258] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:21:24.853928 1028 ts_tablet_manager.cc:1905] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:21:24.855139 1031 ts_tablet_manager.cc:1905] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:21:24.860754 1029 ts_tablet_manager.cc:1905] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:21:24.867520 1031 ts_tablet_manager.cc:1918] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:21:24.868213 1031 log.cc:1196] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/82ac8931371a43d2bffa32f767bf91b3
I20260810 03:21:24.869366 1031 ts_tablet_manager.cc:1939] T 82ac8931371a43d2bffa32f767bf91b3 P 6c0dca9320854598b2307e1bb438f94b: Deleting consensus metadata
I20260810 03:21:24.873868 1029 ts_tablet_manager.cc:1918] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:21:24.873811 436 catalog_manager.cc:4841] TS 6c0dca9320854598b2307e1bb438f94b (127.19.246.194:45281): tablet 82ac8931371a43d2bffa32f767bf91b3 (table test-table [id=68ffec01160c421b9b26daff76bc114b]) successfully deleted
I20260810 03:21:24.874203 1029 log.cc:1196] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/82ac8931371a43d2bffa32f767bf91b3
I20260810 03:21:24.875206 1029 ts_tablet_manager.cc:1939] T 82ac8931371a43d2bffa32f767bf91b3 P 20cae58e1a9148b3abc7bcec6c637f4d: Deleting consensus metadata
I20260810 03:21:24.878604 436 catalog_manager.cc:4841] TS 20cae58e1a9148b3abc7bcec6c637f4d (127.19.246.193:39627): tablet 82ac8931371a43d2bffa32f767bf91b3 (table test-table [id=68ffec01160c421b9b26daff76bc114b]) successfully deleted
I20260810 03:21:24.879035 1028 ts_tablet_manager.cc:1918] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:21:24.879441 1028 log.cc:1196] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/82ac8931371a43d2bffa32f767bf91b3
Aug 10 03:21:24 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:21:24.880676 1028 ts_tablet_manager.cc:1939] T 82ac8931371a43d2bffa32f767bf91b3 P ecf54dcf0edc4aa98797a485574c20cf: Deleting consensus metadata
I20260810 03:21:24.883869 432 catalog_manager.cc:4841] TS ecf54dcf0edc4aa98797a485574c20cf (127.19.246.195:43785): tablet 82ac8931371a43d2bffa32f767bf91b3 (table test-table [id=68ffec01160c421b9b26daff76bc114b]) successfully deleted
I20260810 03:21:24.892338 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:24.848056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.19.246.193:39627)
0810 03:21:24.848393 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:24.848400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:24.859522 (+ 11122us) server_negotiation.cc:407] Connection header received
0810 03:21:24.861491 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:24.861500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:24.861600 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:24.861685 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:21:24.864195 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.865682 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.870253 (+ 4571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:24.871848 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:24.872097 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:24.881456 (+ 9359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:21:24.881475 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:21:24.881492 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:21:24.881536 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:21:24.887686 (+ 6150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:24.888702 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:24.888712 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:24.888714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:24.888967 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:24.889511 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:24.889515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:24.889517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:24.890192 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:21:24.890841 (+ 649us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:24.891208 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 03:21:24.891619 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 03:21:24.893388 678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:21:25.065923 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 537
I20260810 03:21:25.099339 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 691
Aug 10 03:21:25 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:21:25.121724 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:25.100363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.19.246.194:45281)
0810 03:21:25.100567 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:25.100574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:25.100597 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:21:25.100730 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:25.100736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:25.100815 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:25.100937 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:21:25.102774 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:25.103621 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:25.105725 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:25.106228 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:25.106399 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:25.112956 (+ 6557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:21:25.112976 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:21:25.112993 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:21:25.113041 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:21:25.116865 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:25.117802 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:25.117811 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:25.117813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:25.117971 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:25.118788 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:25.118795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:25.118798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:25.119611 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:21:25.120115 (+ 504us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:25.120429 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 03:21:25.120839 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:21:25.122700 834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:21:25.245398 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 691
I20260810 03:21:25.270596 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 846
Aug 10 03:21:25 dist-test-slave-btw8 krb5kdc[32394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:21:25.296475 1036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:21:25.271888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.19.246.195:43785)
0810 03:21:25.272625 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0810 03:21:25.272630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:21:25.272648 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:21:25.272780 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:21:25.272787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:21:25.272862 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:21:25.272959 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:21:25.275148 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:25.276427 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:25.278674 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:21:25.279530 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:21:25.279768 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:21:25.287433 (+ 7665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:21:25.287462 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:21:25.287480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:21:25.287543 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:21:25.292016 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:25.293428 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:25.293437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:25.293440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:25.293859 (+ 419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:21:25.294422 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:21:25.294426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:21:25.294428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:21:25.294957 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:21:25.295232 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 03:21:25.295723 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 03:21:25.295961 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":421,"threads_started":1}
I20260810 03:21:25.297562 975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:21:25.443456 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 846
I20260810 03:21:25.471411 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 415
I20260810 03:21:25.472954 484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:21:25.607101 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 415
I20260810 03:21:25.637127 20443 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 03:21:25.985446 20443 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 03:21:25.985919 20443 mini_ranger.cc:66] Stopping Ranger...
I20260810 03:21:26.344045 20443 mini_ranger.cc:68] Stopped Ranger
2026-08-10 03:21:26.344 UTC [32429] LOG: received smart shutdown request
2026-08-10 03:21:26.349 UTC [32429] LOG: background worker "logical replication launcher" (PID 32438) exited with exit code 1
2026-08-10 03:21:26.349 UTC [32433] LOG: shutting down
2026-08-10 03:21:26.675 UTC [32429] LOG: database system is shut down
2026-08-10T03:21:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (64453 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:21:26 dist-test-slave-btw8 krb5kdc[1049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:21:26 dist-test-slave-btw8 krb5kdc[1049](info): set up 2 sockets
Aug 10 03:21:26 dist-test-slave-btw8 krb5kdc[1049](info): commencing operation
krb5kdc: starting...
W20260810 03:21:28.107013 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.011s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:21:28 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:21:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/rangeradmin_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/rangeradmin_127.19.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/rangerlookup_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/rangerlookup_127.19.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/HTTP_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/HTTP_127.19.246.212@KRBTEST.COM.keytab.
I20260810 03:21:28.576709 20443 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZHDeAK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/postgres -l logfile start
2026-08-10 03:21:32.109 UTC [1082] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 03:21:32.109 UTC [1082] LOG: listening on IPv4 address "127.19.246.212", port 33251
2026-08-10 03:21:32.115 UTC [1082] LOG: listening on Unix socket "/tmp/.s.PGSQL.33251"
2026-08-10 03:21:32.147 UTC [1085] LOG: database system was shut down at 2026-08-10 03:21:29 UTC
2026-08-10 03:21:32.152 UTC [1082] LOG: database system is ready to accept connections
I20260810 03:21:33.281718 20443 mini_postgres.cc:96] Postgres bound to 33251
2026-08-10 03:21:33.306 UTC [1094] FATAL: database "slave" does not exist
127.19.246.212:33251 - accepting connections
I20260810 03:21:33.307423 20443 mini_ranger.cc:161] Starting Ranger...
I20260810 03:21:33.344440 20443 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 03:21:34.110678 20443 mini_ranger.cc:87] Created ranger Postgres database
I20260810 03:21:34.110837 20443 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-admin
2026-08-10 03:21:34,446 [I] DB FLAVOR :POSTGRES
2026-08-10 03:21:34,447 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:21:34,447 [I] Checking connection..
2026-08-10 03:21:34,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 03:21:34,756 [I] Checking connection passed.
2026-08-10 03:21:34,756 [I] --------- Verifying version history table ---------
2026-08-10 03:21:34,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:21:35,043 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:21:35,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:21:35,327 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:21:35,327 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 03:21:35,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 03:21:35.614 UTC [1162] WARNING: there is no transaction in progress
2026-08-10 03:21:35,625 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 03:21:35,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:21:35,904 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 03:21:35,905 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:21:35,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 03:21:36,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 03:21:36,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-btw8.c.gcp-upstream.internal','N') ;"
2026-08-10 03:21:36,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:21:37,034 [I] Table x_portal_user does not exist in database ranger
2026-08-10 03:21:37,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:21:37,325 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 03:21:37,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:21:37,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:21:37,911 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 03:21:37,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 03:21:39.198 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.225 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.251 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.276 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.297 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.324 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.396 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.406 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.426 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.440 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.455 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.472 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.486 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.494 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.504 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:39.515 UTC [1277] WARNING: there is no transaction in progress
2026-08-10 03:21:40,557 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 03:21:40,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 03:21:40,856 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 03:21:40,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:21:41,173 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 03:21:41,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:21:41,457 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 03:21:41,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:21:41,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:21:42,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-btw8.c.gcp-upstream.internal';"
2026-08-10 03:21:42,381 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 03:21:42,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 03:21:42,695 [I] DB_PATCHES have already been applied
I20260810 03:21:42.702231 20443 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZHDeAK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 03:21:42.702348 20443 mini_ranger.cc:193] Using host: 127.19.246.212
I20260810 03:21:42.724961 20443 mini_ranger.cc:227] Ranger admin URL: http://127.19.246.212:45247
Aug 10, 2026 3:21:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:21:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:21:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:21:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:21:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:21:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:21:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 03:21:44.059834 20443 mini_ranger.cc:160] Time spent starting Ranger: real 10.752s user 0.001s sys 0.069s
Aug 10, 2026 3:22:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:22:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 3:22:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:22:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:22:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 03:22:08.795136 20443 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.246.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.246.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/rangerkms_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.246.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/rangerkms_127.19.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.246.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.246.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.246.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/HTTP_127.19.246.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.246.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/HTTP_127.19.246.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 03:22:09 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332129, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 03:22:09.048060 20443 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 03:22:09.078642 20443 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 03:22:10.177208 20443 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 03:22:10.177320 20443 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms
I20260810 03:22:10.177362 20443 mini_ranger_kms.cc:227] Using postgres at 127.19.246.212:33251
2026-08-10 03:22:10,725 [I] DB FLAVOR :POSTGRES
2026-08-10 03:22:10,725 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:22:10,725 [I] Checking connection
2026-08-10 03:22:10,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 03:22:11,001 [I] connection success
2026-08-10 03:22:11,001 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 03:22:11,001 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 03:22:11,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 03:22:11,266 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 03:22:11,267 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:22:11,267 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 03:22:11,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.246.212:33251/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 03:22:11,587 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 03:22:12.171865 20443 mini_ranger_kms.cc:310] Created kms service
I20260810 03:22:12.394671 20443 mini_ranger_kms.cc:326] Created kudu user
I20260810 03:22:12.450297 20443 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 03:22:12 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332129, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:12.899111 20443 mini_ranger_kms.cc:384] Added ranger policy
I20260810 03:22:12.899518 20443 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZHDeAK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZHDeAK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 03:22:12.899571 20443 mini_ranger_kms.cc:240] Using host: 127.19.246.212
I20260810 03:22:12.924059 20443 mini_ranger_kms.cc:276] Ranger KMS PID: 1473
I20260810 03:22:12.924203 20443 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.246.212:37867
Aug 10, 2026 3:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:22:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 03:22:15 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332135, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.246.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:15.542371 20443 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.494s user 0.066s sys 0.524s
I20260810 03:22:15.542652 20443 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 03:22:15.542843 20443 mini_ranger_kms.cc:401] 127.19.246.212:37867/kms/v1/keys
Aug 10 03:22:16 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332135, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.246.212@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
Aug 10 03:22:16 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332135, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.246.212@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
Aug 10 03:22:19 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:22:19.291857 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:35645
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:35645
--ranger_config_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:37867/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:19.675467 1528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:19.675827 1528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:19.675994 1528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:19.684226 1528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:19.684368 1528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:19.684453 1528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:19.684562 1528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:19.684648 1528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:19.695843 1528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.246.212:37867/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:35645
--ranger_config_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:35645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:19.697268 1528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:19.698805 1528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:19.712636 1536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:19.712587 1537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:19.713212 1528 server_base.cc:1034] running on GCE node
W20260810 03:22:19.713363 1539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:19.714356 1528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:19.716122 1528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:19.717391 1528 hybrid_clock.cc:648] HybridClock initialized: now 1786332139717345 us; error 48 us; skew 500 ppm
I20260810 03:22:19.717808 1528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:19 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332139, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:19.721415 1528 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:22:19.723248 1528 webserver.cc:460] Webserver started at http://127.19.246.254:39697/ using document root <none> and password file <none>
I20260810 03:22:19.724001 1528 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:19.724159 1528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:19.724507 1528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:22:19 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332139, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:19.794611 1528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "5c4e1ad618d447258974edc508e75022"
format_stamp: "Formatted at 2026-08-10 03:22:19 on dist-test-slave-btw8"
server_key: "1a5d72ddbb83fa088ccb1b88767d5311"
server_key_iv: "cbe1de6be724c422fc69be8c68fab016"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:19.795737 1528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "5c4e1ad618d447258974edc508e75022"
format_stamp: "Formatted at 2026-08-10 03:22:19 on dist-test-slave-btw8"
server_key: "1a5d72ddbb83fa088ccb1b88767d5311"
server_key_iv: "cbe1de6be724c422fc69be8c68fab016"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:19.802064 1528 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:22:19.823611 1546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:19.825314 1528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:22:19.825501 1528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "5c4e1ad618d447258974edc508e75022"
format_stamp: "Formatted at 2026-08-10 03:22:19 on dist-test-slave-btw8"
server_key: "1a5d72ddbb83fa088ccb1b88767d5311"
server_key_iv: "cbe1de6be724c422fc69be8c68fab016"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:19.825718 1528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:19.838639 1528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:19.843009 1528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:19.843453 1528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:19.864346 1528 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:35645
I20260810 03:22:19.864367 1607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:35645 every 8 connection(s)
I20260810 03:22:19.865847 1528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:22:19.866427 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 1528
I20260810 03:22:19.866739 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:22:19.867707 20443 external_mini_cluster.cc:1428] Setting key 307758f791a9d022a6e131a25c57793b
I20260810 03:22:19.870575 1609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:19.882371 1609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Bootstrap starting.
Aug 10 03:22:19 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:19.886153 1609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:19.887316 1609 log.cc:824] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:19.891187 1609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: No bootstrap required, opened a new log
I20260810 03:22:19.892815 1612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:19.869852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.19.246.254:35645)
0810 03:22:19.870729 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:19.870749 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:19.870806 (+ 57us) server_negotiation.cc:407] Connection header received
0810 03:22:19.873000 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:19.873023 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:19.873447 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:19.873884 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:19.875417 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:19.876879 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:19.878275 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:19.879026 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:19.879279 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:19.883845 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:19.883858 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:19.883874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:19.883925 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:19.887539 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:19.888620 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:19.888627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:19.888634 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:19.888827 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:19.889255 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:19.889260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:19.889264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:19.889968 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:19.890293 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:19.890769 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 03:22:19.891216 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":390,"threads_started":1}
I20260810 03:22:19.900805 1609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } }
I20260810 03:22:19.901512 1609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:19.901654 1609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4e1ad618d447258974edc508e75022, State: Initialized, Role: FOLLOWER
I20260810 03:22:19.902467 1609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } }
I20260810 03:22:19.902871 1609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:22:19.903048 1609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:22:19.903290 1609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:19.906829 1609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } }
I20260810 03:22:19.907568 1609 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c4e1ad618d447258974edc508e75022; no voters:
I20260810 03:22:19.908469 1609 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:22:19.908600 1614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:22:19.911546 1614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 1 LEADER]: Becoming Leader. State: Replica: 5c4e1ad618d447258974edc508e75022, State: Running, Role: LEADER
I20260810 03:22:19.912154 1614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } }
I20260810 03:22:19.912338 1609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:19.915197 1615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c4e1ad618d447258974edc508e75022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } } }
I20260810 03:22:19.915768 1615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:19.915635 1616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4e1ad618d447258974edc508e75022. Latest consensus state: current_term: 1 leader_uuid: "5c4e1ad618d447258974edc508e75022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } } }
I20260810 03:22:19.915964 1616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:19.916802 1620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:22:19.922852 1620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:22:19.933138 1609 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:22:19.936981 1620 catalog_manager.cc:1347] Generated new cluster ID: ef521e38e74b434eb835701233cea604
I20260810 03:22:19.937155 1620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:22:19.981603 1620 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:22:19.983268 1620 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:22:19.999948 1620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Generated new TSK 0
I20260810 03:22:20.001294 1620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:22:20.836428 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:37867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35645
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:22:21 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:22:21.375072 1655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:21.375497 1655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:21.375726 1655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:21.386164 1655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:21.386368 1655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:21.386585 1655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:22:21.400827 1655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.246.212:37867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:21.402784 1655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:21.404713 1655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:21.423300 1666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:21.423327 1667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:21.424396 1669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:21.424718 1655 server_base.cc:1034] running on GCE node
I20260810 03:22:21.425473 1655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:21.426653 1655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:21.427923 1655 hybrid_clock.cc:648] HybridClock initialized: now 1786332141427866 us; error 43 us; skew 500 ppm
I20260810 03:22:21.428290 1655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:21 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:21.438964 1655 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:22:21.458119 1655 webserver.cc:460] Webserver started at http://127.19.246.193:40159/ using document root <none> and password file <none>
I20260810 03:22:21.459415 1655 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:21.459610 1655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:21.460047 1655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:22:21 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:21.485088 1655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "49ce64b918d6429d949afe2831bacd99"
format_stamp: "Formatted at 2026-08-10 03:22:21 on dist-test-slave-btw8"
server_key: "6493b002ff902f5e6483dde6e84519b9"
server_key_iv: "d842b0f08304884a74bec1402981156c"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:21.486913 1655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "49ce64b918d6429d949afe2831bacd99"
format_stamp: "Formatted at 2026-08-10 03:22:21 on dist-test-slave-btw8"
server_key: "6493b002ff902f5e6483dde6e84519b9"
server_key_iv: "d842b0f08304884a74bec1402981156c"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:21.505975 1655 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.000s sys 0.014s
I20260810 03:22:21.541953 1678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:21.546808 1655 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.007s sys 0.006s
I20260810 03:22:21.547082 1655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "49ce64b918d6429d949afe2831bacd99"
format_stamp: "Formatted at 2026-08-10 03:22:21 on dist-test-slave-btw8"
server_key: "6493b002ff902f5e6483dde6e84519b9"
server_key_iv: "d842b0f08304884a74bec1402981156c"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:21.547434 1655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:21.589988 1655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:21.596273 1655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:21.596731 1655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:21.608817 1655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Aug 10 03:22:21 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:21.617504 1655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:21.617739 1655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 03:22:21.617923 1655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:21.618033 1655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:21.754519 1655 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:34509
I20260810 03:22:21.754643 1800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:34509 every 8 connection(s)
I20260810 03:22:21.756572 1655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:22:21.758177 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 1655
I20260810 03:22:21.758425 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:22:21.759335 20443 external_mini_cluster.cc:1428] Setting key 4eb99a28d5ba05744ea9f7ccc26f3393
Aug 10 03:22:21 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
I20260810 03:22:21.817487 1803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:21.763030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:37163 (local address 127.19.246.254:35645)
0810 03:22:21.789324 (+ 26294us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:21.789333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:21.789359 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:22:21.792035 (+ 2676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:21.792043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:21.792168 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:21.792283 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:21.797324 (+ 5041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:21.798686 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:21.801685 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:21.802515 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:21.802661 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:21.808432 (+ 5771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:21.808442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:21.808450 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:21.808508 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:21.812993 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:21.814276 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:21.814284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:21.814287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:21.814482 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:21.815171 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:21.815177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:21.815180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:21.815600 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:21.815752 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:21.816812 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0810 03:22:21.817072 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26148,"thread_start_us":25999,"threads_started":1}
I20260810 03:22:21.820083 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:21.763792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35645 (local address 127.19.246.193:37163)
0810 03:22:21.788360 (+ 24568us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:21.788432 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:21.791612 (+ 3180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:21.795185 (+ 3573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:21.795203 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:21.796037 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:21.797021 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:21.797055 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:21.798954 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:21.798962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:21.801383 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:21.801419 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:21.803087 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:21.803097 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:21.803501 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:21.804500 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:21.804538 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:21.807997 (+ 3459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:21.813326 (+ 5329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:21.813340 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:21.813365 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:21.814082 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:21.814644 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:21.814649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:21.814653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:21.815010 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:21.815835 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:21.815843 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:21.816337 (+ 494us) client_negotiation.cc:769] Sending connection context
0810 03:22:21.817927 (+ 1590us) client_negotiation.cc:240] Negotiation successful
0810 03:22:21.818563 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24169,"thread_start_us":23878,"threads_started":1}
I20260810 03:22:21.822389 1801 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35645
I20260810 03:22:21.822736 1801 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:21.823670 1801 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
I20260810 03:22:21.826442 1568 ts_manager.cc:194] Registered new tserver with Master: 49ce64b918d6429d949afe2831bacd99 (127.19.246.193:34509)
I20260810 03:22:21.828747 1568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:37163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:22:21.859869 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:37867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35645
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:22:22 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
W20260810 03:22:22.351670 1807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:22.352074 1807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:22.352263 1807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:22.360277 1807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:22.360458 1807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:22.360675 1807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:22:22.371198 1807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.246.212:37867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:22.372879 1807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:22.374285 1807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:22.389909 1828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:22.389950 1825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:22.389875 1826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:22.390832 1807 server_base.cc:1034] running on GCE node
I20260810 03:22:22.391651 1807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:22.392804 1807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:22.394030 1807 hybrid_clock.cc:648] HybridClock initialized: now 1786332142393947 us; error 80 us; skew 500 ppm
I20260810 03:22:22.394424 1807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:22 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332142, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:22.398069 1807 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:22:22.399956 1807 webserver.cc:460] Webserver started at http://127.19.246.194:39987/ using document root <none> and password file <none>
I20260810 03:22:22.400723 1807 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:22.400846 1807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:22.401139 1807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:22:22 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332142, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:22.420804 1807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "13faa5977b464174a6a30d87405505ca"
format_stamp: "Formatted at 2026-08-10 03:22:22 on dist-test-slave-btw8"
server_key: "9a6a80980d0cd86ea59949393c1c099d"
server_key_iv: "64bda8b01e2efbb9f19de0dad551f8d8"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:22.422135 1807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "13faa5977b464174a6a30d87405505ca"
format_stamp: "Formatted at 2026-08-10 03:22:22 on dist-test-slave-btw8"
server_key: "9a6a80980d0cd86ea59949393c1c099d"
server_key_iv: "64bda8b01e2efbb9f19de0dad551f8d8"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:22.430637 1807 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260810 03:22:22.452893 1837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:22.455309 1807 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260810 03:22:22.455598 1807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "13faa5977b464174a6a30d87405505ca"
format_stamp: "Formatted at 2026-08-10 03:22:22 on dist-test-slave-btw8"
server_key: "9a6a80980d0cd86ea59949393c1c099d"
server_key_iv: "64bda8b01e2efbb9f19de0dad551f8d8"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:22.455924 1807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:22.478672 1807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:22.484572 1807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:22.484953 1807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:22.486311 1807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:22.488392 1807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:22.488479 1807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:22.488618 1807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:22.488692 1807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:22.538543 1807 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:34987
I20260810 03:22:22.538545 1973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:34987 every 8 connection(s)
I20260810 03:22:22.540417 1807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:22:22.550849 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 1807
I20260810 03:22:22.551070 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:22:22.551731 20443 external_mini_cluster.cc:1428] Setting key b040aab22726f2448fb36313163623b7
Aug 10 03:22:22 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332142, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:22.567380 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:22.543649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:36537 (local address 127.19.246.254:35645)
0810 03:22:22.544207 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:22.544214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:22.545530 (+ 1316us) server_negotiation.cc:407] Connection header received
0810 03:22:22.547508 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:22.547516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:22.547627 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:22.547766 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:22.549370 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:22.550594 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:22.553131 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:22.553786 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:22.553946 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:22.558839 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:22.558847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:22.558852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:22.558898 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:22.563012 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:22.564146 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:22.564155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:22.564161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:22.564335 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:22.564912 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:22.564920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:22.564925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:22.565344 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:22.565517 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:22.566514 (+ 997us) server_negotiation.cc:299] Negotiation successful
0810 03:22:22.566812 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":418,"thread_start_us":268,"threads_started":1}
I20260810 03:22:22.568579 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:22.544371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35645 (local address 127.19.246.194:36537)
0810 03:22:22.545316 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:22.545365 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:22.547122 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:22.547994 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:22.548003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:22.548472 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:22.549137 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:22.549155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:22.550839 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:22.550845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:22.552965 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:22.552983 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:22.554076 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:22.554084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:22.554418 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:22.555368 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:22.555404 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:22.558505 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:22.563188 (+ 4683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:22.563199 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:22.563219 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:22.563964 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:22.564489 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:22.564494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:22.564498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:22.564772 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:22.565515 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:22.565524 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:22.565990 (+ 466us) client_negotiation.cc:769] Sending connection context
0810 03:22:22.566332 (+ 342us) client_negotiation.cc:240] Negotiation successful
0810 03:22:22.566905 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":437,"threads_started":1}
I20260810 03:22:22.570441 1974 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35645
I20260810 03:22:22.570811 1974 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:22.571852 1974 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
I20260810 03:22:22.573853 1568 ts_manager.cc:194] Registered new tserver with Master: 13faa5977b464174a6a30d87405505ca (127.19.246.194:34987)
I20260810 03:22:22.575001 1568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:36537
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:22:22.628376 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:37867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35645
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:22:22.833256 1801 heartbeater.cc:503] Master 127.19.246.254:35645 was elected leader, sending a full tablet report...
W20260810 03:22:23.020288 1980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:23.020617 1980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:23.020720 1980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:23.028374 1980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:23.028515 1980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:23.028654 1980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:22:23.040488 1980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.246.212:37867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:23.041986 1980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:23.043565 1980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:23.058393 1987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:23.058370 1989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:23.058423 1986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:23.059199 1980 server_base.cc:1034] running on GCE node
I20260810 03:22:23.060072 1980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:23.061374 1980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:23.062660 1980 hybrid_clock.cc:648] HybridClock initialized: now 1786332143062639 us; error 107 us; skew 500 ppm
I20260810 03:22:23.063191 1980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:23 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:23.066892 1980 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:22:23.068708 1980 webserver.cc:460] Webserver started at http://127.19.246.195:45311/ using document root <none> and password file <none>
I20260810 03:22:23.069439 1980 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:23.069555 1980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:23.069897 1980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:22:23 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:23.088107 1980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "c33882cb5bb04435973310f770025c22"
format_stamp: "Formatted at 2026-08-10 03:22:23 on dist-test-slave-btw8"
server_key: "407e59f9d0002e7b1cda710cb82d76cd"
server_key_iv: "8335b3b5bda6d7773c040d8f153d3210"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:23.089356 1980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "c33882cb5bb04435973310f770025c22"
format_stamp: "Formatted at 2026-08-10 03:22:23 on dist-test-slave-btw8"
server_key: "407e59f9d0002e7b1cda710cb82d76cd"
server_key_iv: "8335b3b5bda6d7773c040d8f153d3210"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:23.097241 1980 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260810 03:22:23.114367 1997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:23.116489 1980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 03:22:23.116734 1980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "c33882cb5bb04435973310f770025c22"
format_stamp: "Formatted at 2026-08-10 03:22:23 on dist-test-slave-btw8"
server_key: "407e59f9d0002e7b1cda710cb82d76cd"
server_key_iv: "8335b3b5bda6d7773c040d8f153d3210"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:23.117033 1980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:23.139897 1980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:23.144760 1980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:23.145098 1980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:23.146221 1980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:23.147845 1980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:23.147961 1980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:23.148120 1980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:23.148203 1980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:23.186404 1980 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:45397
I20260810 03:22:23.186451 2122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:45397 every 8 connection(s)
I20260810 03:22:23.187852 1980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:22:23.193454 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 1980
I20260810 03:22:23.193733 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:22:23.194286 20443 external_mini_cluster.cc:1428] Setting key 6a5473d3fa2a045136f05b2692075ce7
Aug 10 03:22:23 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332143, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:23.210971 2125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:23.190684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:55029 (local address 127.19.246.254:35645)
0810 03:22:23.191294 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:23.191302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:23.192350 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 03:22:23.194451 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:23.194457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:23.194562 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:23.194703 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:23.196422 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.197352 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.199073 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.199610 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.199784 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:23.203711 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:23.203717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:23.203721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:23.203759 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:23.206817 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:23.207984 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:23.207990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:23.207992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:23.208147 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:23.208811 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:23.208816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:23.208819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:23.209145 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:23.209356 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:23.210132 (+ 776us) server_negotiation.cc:299] Negotiation successful
0810 03:22:23.210396 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":253,"threads_started":1}
I20260810 03:22:23.212183 2126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:23.191283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35645 (local address 127.19.246.195:55029)
0810 03:22:23.192162 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:23.192210 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:23.194137 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:23.194968 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:23.194977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:23.195565 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:23.196242 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:23.196261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.197547 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.197557 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:23.198891 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:23.198915 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.199902 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.199911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:23.200173 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:23.200937 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:23.200962 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:23.203409 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:23.207041 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:23.207051 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:23.207067 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:23.207817 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:23.208334 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:23.208340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:23.208344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:23.208616 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:23.209318 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:23.209328 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:23.209719 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 03:22:23.210024 (+ 305us) client_negotiation.cc:240] Negotiation successful
0810 03:22:23.210610 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":332,"threads_started":1}
I20260810 03:22:23.213936 2123 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35645
I20260810 03:22:23.214306 2123 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:23.215142 2123 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
I20260810 03:22:23.216851 1568 ts_manager.cc:194] Registered new tserver with Master: c33882cb5bb04435973310f770025c22 (127.19.246.195:45397)
I20260810 03:22:23.217756 1568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:55029
I20260810 03:22:23.224320 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:22:23.226154 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 1655
Aug 10 03:22:23 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:22:23.246265 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:23.227195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.19.246.193:34509)
0810 03:22:23.228010 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:23.228018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:23.228047 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:22:23.228113 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:23.228120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:23.228349 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:23.228500 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:23.230611 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.232072 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.233799 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.234425 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.234548 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:23.239784 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:23.239803 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:23.239821 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:23.239901 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:23.242886 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:23.243985 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:23.243994 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:23.243996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:23.244190 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:23.244682 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:23.244686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:23.244688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:23.245090 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:23.245350 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:23.245688 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 03:22:23.245969 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":467,"threads_started":1}
I20260810 03:22:23.247769 1795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:23.337788 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 1655
I20260810 03:22:23.358300 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 1807
Aug 10 03:22:23 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:22:23.382912 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:23.359341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.19.246.194:34987)
0810 03:22:23.360063 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:23.360072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:23.360101 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:22:23.360168 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:23.360174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:23.360402 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:23.360556 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:23.362156 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.363466 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.364972 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.365614 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.365723 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:23.376177 (+ 10454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:23.376197 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:23.376214 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:23.376274 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:23.379575 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:23.380633 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:23.380641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:23.380643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:23.380842 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:23.381326 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:23.381329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:23.381331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:23.381757 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:23.381986 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:23.382302 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 03:22:23.382522 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":368,"threads_started":1}
I20260810 03:22:23.384478 1966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:23.473598 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 1807
I20260810 03:22:23.493650 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 1980
Aug 10 03:22:23 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:22:23.512948 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:23.494712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.19.246.195:45397)
0810 03:22:23.495470 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:23.495478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:23.495509 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:22:23.495574 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:23.495581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:23.495928 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:23.496085 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:23.497663 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.498761 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.500572 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:23.501311 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:23.501426 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:23.506540 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:23.506556 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:23.506572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:23.506624 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:23.509786 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:23.510613 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:23.510621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:23.510624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:23.510773 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:23.511183 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:23.511187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:23.511189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:23.511687 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:23.511953 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:23.512319 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 03:22:23.512534 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":375,"threads_started":1}
I20260810 03:22:23.514382 2118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:23.602155 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 1980
I20260810 03:22:23.621528 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 1528
I20260810 03:22:23.622951 1602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:23.734777 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 1528
I20260810 03:22:23.762851 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:35645
--webserver_interface=127.19.246.254
--webserver_port=39697
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:35645
--ranger_config_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:37867/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.246.212:37867/kms with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:24.141881 2136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:24.142292 2136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:24.142457 2136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:24.150188 2136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:24.150381 2136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:24.150496 2136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:24.150604 2136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:24.150686 2136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:24.162909 2136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.246.212:37867/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:35645
--ranger_config_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:35645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:24.164428 2136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:24.165992 2136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:24.180006 2141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:24.179888 2142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:24.179970 2144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:24.181386 2136 server_base.cc:1034] running on GCE node
I20260810 03:22:24.182441 2136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:24.184288 2136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:24.185576 2136 hybrid_clock.cc:648] HybridClock initialized: now 1786332144185520 us; error 42 us; skew 500 ppm
I20260810 03:22:24.186039 2136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:24 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332144, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:24.189611 2136 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:22:24.191432 2136 webserver.cc:460] Webserver started at http://127.19.246.254:39697/ using document root <none> and password file <none>
I20260810 03:22:24.192183 2136 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:24.192365 2136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:22:24 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332144, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:24.283778 2136 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:22:24.287652 2154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:24.289430 2136 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:22:24.289697 2136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "5c4e1ad618d447258974edc508e75022"
format_stamp: "Formatted at 2026-08-10 03:22:19 on dist-test-slave-btw8"
server_key: "1a5d72ddbb83fa088ccb1b88767d5311"
server_key_iv: "cbe1de6be724c422fc69be8c68fab016"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:24.290802 2136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:24.314087 2136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:24.319522 2136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:24.319875 2136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:24.341336 2136 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:35645
I20260810 03:22:24.341360 2211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:35645 every 8 connection(s)
I20260810 03:22:24.343045 2136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:22:24.347710 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 2136
I20260810 03:22:24.348878 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:37867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:34509
--local_ip_for_outbound_sockets=127.19.246.193
--tserver_master_addrs=127.19.246.254:35645
--webserver_port=40159
--webserver_interface=127.19.246.193
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.246.212:37867/kms with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:22:24.352192 2212 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:22:24.357856 2212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Bootstrap starting.
I20260810 03:22:24.379947 2212 log.cc:824] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:24.390074 2212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:22:24.390717 2212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Bootstrap complete.
I20260810 03:22:24.401017 2212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } }
I20260810 03:22:24.402030 2212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4e1ad618d447258974edc508e75022, State: Initialized, Role: FOLLOWER
I20260810 03:22:24.402833 2212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } }
I20260810 03:22:24.403111 2212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:22:24.403257 2212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:22:24.403484 2212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:22:24.407797 2212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } }
I20260810 03:22:24.408545 2212 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c4e1ad618d447258974edc508e75022; no voters:
I20260810 03:22:24.409512 2212 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:22:24.409679 2217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:22:24.411387 2217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [term 2 LEADER]: Becoming Leader. State: Replica: 5c4e1ad618d447258974edc508e75022, State: Running, Role: LEADER
I20260810 03:22:24.412137 2217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } }
I20260810 03:22:24.412362 2212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:24.415271 2218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5c4e1ad618d447258974edc508e75022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } } }
I20260810 03:22:24.415839 2218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:24.420495 2219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4e1ad618d447258974edc508e75022. Latest consensus state: current_term: 2 leader_uuid: "5c4e1ad618d447258974edc508e75022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e1ad618d447258974edc508e75022" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35645 } } }
I20260810 03:22:24.421797 2219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:24.423892 2223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:22:24.431098 2223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:22:24.433094 2223 catalog_manager.cc:1259] Loaded cluster ID: ef521e38e74b434eb835701233cea604
I20260810 03:22:24.433300 2223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:22:24.437841 2223 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:22:24.439858 2223 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Loaded TSK: 0
I20260810 03:22:24.441844 2223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:22:24.444931 2212 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 03:22:24.862813 2214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:24.863415 2214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:24.863592 2214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:24.876235 2214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:24.876489 2214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:24.876763 2214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:22:24.893451 2214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.246.212:37867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:34509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=40159
--tserver_master_addrs=127.19.246.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:24.895481 2214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:24.897598 2214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:24.930464 2254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:24.937716 2253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:24.938788 2256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:24.939414 2214 server_base.cc:1034] running on GCE node
I20260810 03:22:24.940301 2214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:24.941692 2214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:24.943038 2214 hybrid_clock.cc:648] HybridClock initialized: now 1786332144942961 us; error 72 us; skew 500 ppm
I20260810 03:22:24.943516 2214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:24 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332144, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:24.948637 2214 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:22:24.956015 2214 webserver.cc:460] Webserver started at http://127.19.246.193:40159/ using document root <none> and password file <none>
I20260810 03:22:24.957301 2214 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:24.957679 2214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:22:24 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332144, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:25.003985 2214 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.010s sys 0.005s
I20260810 03:22:25.009049 2265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:25.011696 2214 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:22:25.011994 2214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "49ce64b918d6429d949afe2831bacd99"
format_stamp: "Formatted at 2026-08-10 03:22:21 on dist-test-slave-btw8"
server_key: "6493b002ff902f5e6483dde6e84519b9"
server_key_iv: "d842b0f08304884a74bec1402981156c"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:25.013984 2214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:25.061635 2214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:25.068399 2214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:25.068811 2214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:25.070372 2214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:25.072633 2214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:25.072804 2214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:25.072938 2214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:25.073052 2214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:25.135306 2214 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:34509
I20260810 03:22:25.135394 2390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:34509 every 8 connection(s)
I20260810 03:22:25.137442 2214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:22:25.138337 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 2214
I20260810 03:22:25.139182 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:37867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:34987
--local_ip_for_outbound_sockets=127.19.246.194
--tserver_master_addrs=127.19.246.254:35645
--webserver_port=39987
--webserver_interface=127.19.246.194
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.246.212:37867/kms with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:22:25 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332144, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:25.189620 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:25.141510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35645 (local address 127.19.246.193:53563)
0810 03:22:25.156869 (+ 15359us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:25.156927 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:25.160044 (+ 3117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:25.162885 (+ 2841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:25.162899 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:25.163963 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:25.164824 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:25.164855 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:25.167457 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:25.167464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:25.169601 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:25.169636 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:25.171712 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:25.171722 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:25.172066 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:25.173083 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:25.173121 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:25.179272 (+ 6151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:25.184245 (+ 4973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:25.184253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:25.184266 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:25.184974 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:25.185532 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:25.185538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:25.185542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:25.185740 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:25.186832 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:25.186840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:25.187161 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 03:22:25.187466 (+ 305us) client_negotiation.cc:240] Negotiation successful
0810 03:22:25.188113 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15021,"thread_start_us":14742,"threads_started":1}
I20260810 03:22:25.189749 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:25.142745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:53563 (local address 127.19.246.254:35645)
0810 03:22:25.156072 (+ 13327us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:25.156106 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:25.157465 (+ 1359us) server_negotiation.cc:407] Connection header received
0810 03:22:25.160999 (+ 3534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:25.161020 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:25.161585 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:25.162033 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:25.165138 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:25.167124 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:25.169846 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:25.171011 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:25.171224 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:25.179782 (+ 8558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:25.179795 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:25.179821 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:25.179883 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:25.183936 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:25.185181 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:25.185198 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:25.185204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:25.185393 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:25.185881 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:25.185887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:25.185890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:25.186554 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:25.186764 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:25.187710 (+ 946us) server_negotiation.cc:299] Negotiation successful
0810 03:22:25.188113 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12909,"thread_start_us":10593,"threads_started":1}
I20260810 03:22:25.194972 2391 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35645
I20260810 03:22:25.195442 2391 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:25.196552 2391 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
W20260810 03:22:25.199980 2391 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:35645 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:22:25.200263 2391 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:25.200587 2391 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
I20260810 03:22:25.202293 2391 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:25.202807 2391 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
W20260810 03:22:25.204188 2391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:22:25.613528 2395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:25.614121 2395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:25.614319 2395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:25.622419 2395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:25.622596 2395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:25.622785 2395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:22:25.638074 2395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.246.212:37867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:34987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=39987
--tserver_master_addrs=127.19.246.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:25.640273 2395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:25.641965 2395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:25.663511 2411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:25.664711 2412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:25.666260 2414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:25.666260 2395 server_base.cc:1034] running on GCE node
I20260810 03:22:25.667225 2395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:25.671801 2395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:25.673204 2395 hybrid_clock.cc:648] HybridClock initialized: now 1786332145673079 us; error 125 us; skew 500 ppm
I20260810 03:22:25.673816 2395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:25 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332145, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:25.679219 2395 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:22:25.682984 2395 webserver.cc:460] Webserver started at http://127.19.246.194:39987/ using document root <none> and password file <none>
I20260810 03:22:25.684202 2395 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:25.684432 2395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:22:25 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332145, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:25.721690 2395 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.004s sys 0.004s
I20260810 03:22:25.731956 2422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:25.734263 2395 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.001s
I20260810 03:22:25.734588 2395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "13faa5977b464174a6a30d87405505ca"
format_stamp: "Formatted at 2026-08-10 03:22:22 on dist-test-slave-btw8"
server_key: "9a6a80980d0cd86ea59949393c1c099d"
server_key_iv: "64bda8b01e2efbb9f19de0dad551f8d8"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:25.736281 2395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:25.756086 2395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:25.762013 2395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:25.762454 2395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:25.779932 2395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:25.788614 2395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:25.788838 2395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:25.789021 2395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:25.789124 2395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:25.848776 2395 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:34987
I20260810 03:22:25.848776 2548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:34987 every 8 connection(s)
I20260810 03:22:25.850679 2395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
Aug 10 03:22:25 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786332145, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:25.857970 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 2395
I20260810 03:22:25.858822 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.246.212:37867/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:45397
--local_ip_for_outbound_sockets=127.19.246.195
--tserver_master_addrs=127.19.246.254:35645
--webserver_port=45311
--webserver_interface=127.19.246.195
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.246.212:37867/kms with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:22:25 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332145, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:25.887663 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:25.855823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:55103 (local address 127.19.246.254:35645)
0810 03:22:25.861319 (+ 5496us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:25.861329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:25.861361 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:22:25.864361 (+ 3000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:25.864370 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:25.864496 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:25.864615 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:25.867171 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:25.868520 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:25.871298 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:25.873037 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:25.873173 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:25.878500 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:25.878510 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:25.878518 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:25.878571 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:25.882958 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:25.884154 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:25.884164 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:25.884170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:25.884363 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:25.885077 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:25.885085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:25.885090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:25.885524 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:25.885661 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:25.886823 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0810 03:22:25.887137 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5316,"thread_start_us":7796,"threads_started":1}
I20260810 03:22:25.890780 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:25.854892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35645 (local address 127.19.246.194:55103)
0810 03:22:25.856457 (+ 1565us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:25.856506 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:25.859172 (+ 2666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:25.865158 (+ 5986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:25.865174 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:25.865950 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:25.866899 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:25.866932 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:25.868799 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:25.868807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:25.871059 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:25.871081 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:25.873604 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:25.873614 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:25.874000 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:25.874873 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:25.874915 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:25.878160 (+ 3245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:25.883183 (+ 5023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:25.883197 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:25.883221 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:25.883988 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:25.884629 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:25.884637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:25.884643 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:25.884933 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:25.885780 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:25.885792 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:25.886350 (+ 558us) client_negotiation.cc:769] Sending connection context
0810 03:22:25.888322 (+ 1972us) client_negotiation.cc:240] Negotiation successful
0810 03:22:25.889062 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"thread_start_us":976,"threads_started":1}
I20260810 03:22:25.893287 2549 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35645
I20260810 03:22:25.893678 2549 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:25.894794 2549 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
I20260810 03:22:25.897365 2175 ts_manager.cc:194] Registered new tserver with Master: 13faa5977b464174a6a30d87405505ca (127.19.246.194:34987)
I20260810 03:22:25.899434 2175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:55103
I20260810 03:22:26.204353 2391 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:26.204882 2391 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
I20260810 03:22:26.206306 2175 ts_manager.cc:194] Registered new tserver with Master: 49ce64b918d6429d949afe2831bacd99 (127.19.246.193:34509)
I20260810 03:22:26.207878 2175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:53563
Aug 10 03:22:26 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332145, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
W20260810 03:22:26.389304 2553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:26.389915 2553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:26.390165 2553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:26.402448 2553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:26.402729 2553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:26.402979 2553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:22:26.421751 2553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.246.212:37867/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:45397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=45311
--tserver_master_addrs=127.19.246.254:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:26.424300 2553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:26.426638 2553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:26.469837 2561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:26.470680 2564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:26.471329 2553 server_base.cc:1034] running on GCE node
W20260810 03:22:26.472015 2560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:26.473042 2553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:26.494926 2553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:26.498544 2553 hybrid_clock.cc:648] HybridClock initialized: now 1786332146498427 us; error 91 us; skew 500 ppm
I20260810 03:22:26.499104 2553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:26 dist-test-slave-btw8 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332146, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:26.504736 2553 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:22:26.530320 2553 webserver.cc:460] Webserver started at http://127.19.246.195:45311/ using document root <none> and password file <none>
I20260810 03:22:26.531602 2553 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:26.531867 2553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:22:26 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332146, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:26.579389 2553 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.003s sys 0.007s
I20260810 03:22:26.590165 2572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:26.592670 2553 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.000s sys 0.010s
I20260810 03:22:26.593005 2553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "c33882cb5bb04435973310f770025c22"
format_stamp: "Formatted at 2026-08-10 03:22:23 on dist-test-slave-btw8"
server_key: "407e59f9d0002e7b1cda710cb82d76cd"
server_key_iv: "8335b3b5bda6d7773c040d8f153d3210"
server_key_version: "kuduclusterkey@0"
I20260810 03:22:26.594820 2553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:26.662132 2553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:26.668381 2553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:26.668944 2553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:26.677376 2553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:26.686014 2553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:26.686259 2553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 03:22:26.686436 2553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:26.686551 2553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:22:26 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332145, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for HTTP/127.19.246.212@KRBTEST.COM
I20260810 03:22:26.905603 2549 heartbeater.cc:503] Master 127.19.246.254:35645 was elected leader, sending a full tablet report...
I20260810 03:22:27.211792 2391 heartbeater.cc:503] Master 127.19.246.254:35645 was elected leader, sending a full tablet report...
I20260810 03:22:27.254988 2553 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:45397
I20260810 03:22:27.255030 2703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:45397 every 8 connection(s)
I20260810 03:22:27.257097 2553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:22:27.260802 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 2553
Aug 10 03:22:27 dist-test-slave-btw8 krb5kdc[1049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332146, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:27.295943 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.261947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.19.246.254:35645)
0810 03:22:27.277907 (+ 15960us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.277914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.277935 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:22:27.277998 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.278003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.278072 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.278204 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:27.280613 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.281902 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.283892 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.284557 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.284693 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.288844 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:27.288850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:27.288854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:27.288905 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:27.292251 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.293337 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.293345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.293349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.293546 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.294038 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.294043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.294046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.294476 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:27.294807 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.295209 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.295525 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15837,"thread_start_us":14887,"threads_started":1}
I20260810 03:22:27.297752 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.261303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35645 (local address 127.19.246.195:45987)
0810 03:22:27.274232 (+ 12929us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:27.274294 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:27.276375 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:27.277723 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:27.277738 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:27.278510 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:27.279179 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.279199 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.280354 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.280358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.282453 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.282468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.283394 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.283403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.283678 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.284601 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:27.284642 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:27.287441 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:27.292587 (+ 5146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:27.292601 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:27.292626 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:27.293350 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:27.293858 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:27.293863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:27.293868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:27.294178 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:27.294894 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:27.294901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:27.295476 (+ 575us) client_negotiation.cc:769] Sending connection context
0810 03:22:27.295831 (+ 355us) client_negotiation.cc:240] Negotiation successful
0810 03:22:27.296471 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12579,"thread_start_us":12312,"threads_started":1}
I20260810 03:22:27.299008 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.260557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:45987 (local address 127.19.246.254:35645)
0810 03:22:27.277047 (+ 16490us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.277052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.277069 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:22:27.277149 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.277152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.277220 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.277362 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:27.279412 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.280167 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.282619 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.283056 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.283234 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.287726 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:27.287732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:27.287735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:27.287782 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:27.292401 (+ 4619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.293498 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.293504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.293507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.293705 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.294319 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.294324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.294327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.294668 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:27.294807 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.298335 (+ 3528us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.298597 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":16368,"thread_start_us":17049,"threads_started":1}
I20260810 03:22:27.300557 2704 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35645
I20260810 03:22:27.300920 2704 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:27.301904 2704 heartbeater.cc:511] Master 127.19.246.254:35645 requested a full tablet report, sending...
I20260810 03:22:27.303755 2175 ts_manager.cc:194] Registered new tserver with Master: c33882cb5bb04435973310f770025c22 (127.19.246.195:45397)
I20260810 03:22:27.304610 2175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:45987
I20260810 03:22:27.313056 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:22:27.335510 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.317977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.19.246.254:35645)
0810 03:22:27.318281 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.318290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.318753 (+ 463us) server_negotiation.cc:407] Connection header received
0810 03:22:27.318919 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.318925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.319024 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.319154 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:27.321016 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.322185 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.323985 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.324809 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.324969 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.328599 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:27.328606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:27.328611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:27.328660 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:27.332266 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.333162 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.333168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.333172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.333368 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.333777 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.333782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.333786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.334164 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:27.334438 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.334841 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.335118 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 03:22:27.342321 2175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:22:27.352567 2175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:22:27.405654 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.370372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:34509 (local address 127.0.0.1:42774)
0810 03:22:27.383227 (+ 12855us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:27.383296 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:27.385016 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:27.396501 (+ 11485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:27.396507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:27.396541 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:27.396972 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.396985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.399112 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.399117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.401973 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.401994 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.404364 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.404372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.404645 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.404740 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 03:22:27.404870 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 03:22:27.405152 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12502,"mutex_wait_us":29,"thread_start_us":13049,"threads_started":1}
I20260810 03:22:27.405854 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.371444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.19.246.195:45397)
0810 03:22:27.381585 (+ 10141us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.381597 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.393703 (+ 12106us) server_negotiation.cc:407] Connection header received
0810 03:22:27.394152 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.394180 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.394579 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.394848 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:27.396255 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.398062 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.402331 (+ 4269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.404041 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.404285 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.404503 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.405246 (+ 743us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.405480 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9926,"thread_start_us":10789,"threads_started":1}
I20260810 03:22:27.407696 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.370706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.19.246.193:34509)
0810 03:22:27.395491 (+ 24785us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.395504 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.395538 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:22:27.395962 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.395970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.396254 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.397214 (+ 960us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:27.397450 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.398941 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.402336 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.404041 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.406762 (+ 2721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.406974 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.407057 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.407267 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24539,"thread_start_us":24955,"threads_started":1}
I20260810 03:22:27.408366 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.370817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:45397 (local address 127.0.0.1:35670)
0810 03:22:27.393510 (+ 22693us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:27.393541 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:27.393884 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:27.395398 (+ 1514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:27.395407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:27.395444 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:27.395997 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.396013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.399857 (+ 3844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.399865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.402111 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.402124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.404812 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.404818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.405026 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.405057 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:22:27.407752 (+ 2695us) client_negotiation.cc:240] Negotiation successful
0810 03:22:27.407932 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22509,"mutex_wait_us":5,"thread_start_us":22326,"threads_started":1}
I20260810 03:22:27.412510 2318 tablet_service.cc:1467] Processing CreateTablet for tablet cdff41f5c47848af8ae5de7518d714fb (DEFAULT_TABLE table=test-table [id=9e8d33375c6b4aa0a0f01f4a27e5ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:27.412631 2625 tablet_service.cc:1467] Processing CreateTablet for tablet cdff41f5c47848af8ae5de7518d714fb (DEFAULT_TABLE table=test-table [id=9e8d33375c6b4aa0a0f01f4a27e5ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:27.414369 2625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdff41f5c47848af8ae5de7518d714fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:27.414505 2318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdff41f5c47848af8ae5de7518d714fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:27.415160 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.384648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:34987 (local address 127.0.0.1:57110)
0810 03:22:27.405965 (+ 21317us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:27.405989 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:27.406135 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:27.407519 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:27.407524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:27.407555 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:27.407791 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.407803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.410067 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.410074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.412038 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.412115 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.414310 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.414318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.414539 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.414582 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 03:22:27.414661 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:22:27.414841 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21193,"thread_start_us":22541,"threads_started":1}
I20260810 03:22:27.416654 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.384972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.19.246.194:34987)
0810 03:22:27.400529 (+ 15557us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.400542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.406129 (+ 5587us) server_negotiation.cc:407] Connection header received
0810 03:22:27.406302 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.406316 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.406741 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.407984 (+ 1243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:27.408261 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.409887 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.412356 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.414061 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.415867 (+ 1806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.416060 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.416134 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.416313 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15295,"thread_start_us":15592,"threads_started":1}
I20260810 03:22:27.420979 2476 tablet_service.cc:1467] Processing CreateTablet for tablet cdff41f5c47848af8ae5de7518d714fb (DEFAULT_TABLE table=test-table [id=9e8d33375c6b4aa0a0f01f4a27e5ffb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:27.422732 2476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdff41f5c47848af8ae5de7518d714fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:27.439633 2726 tablet_bootstrap.cc:492] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Bootstrap starting.
I20260810 03:22:27.444410 2728 tablet_bootstrap.cc:492] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: Bootstrap starting.
I20260810 03:22:27.446702 2726 tablet_bootstrap.cc:654] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:27.448423 2726 log.cc:824] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:27.450714 2728 tablet_bootstrap.cc:654] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:27.451922 2729 tablet_bootstrap.cc:492] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Bootstrap starting.
I20260810 03:22:27.456931 2728 log.cc:824] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:27.458858 2726 tablet_bootstrap.cc:492] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: No bootstrap required, opened a new log
I20260810 03:22:27.458828 2729 tablet_bootstrap.cc:654] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:27.459496 2726 ts_tablet_manager.cc:1397] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260810 03:22:27.460410 2729 log.cc:824] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:27.471009 2729 tablet_bootstrap.cc:492] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: No bootstrap required, opened a new log
I20260810 03:22:27.471592 2729 ts_tablet_manager.cc:1397] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260810 03:22:27.474318 2726 raft_consensus.cc:348] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.475029 2728 tablet_bootstrap.cc:492] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: No bootstrap required, opened a new log
I20260810 03:22:27.475040 2726 raft_consensus.cc:374] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:27.475139 2726 raft_consensus.cc:729] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33882cb5bb04435973310f770025c22, State: Initialized, Role: FOLLOWER
I20260810 03:22:27.475497 2728 ts_tablet_manager.cc:1397] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.008s
I20260810 03:22:27.476084 2726 consensus_queue.cc:260] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.487164 2728 raft_consensus.cc:348] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.487896 2728 raft_consensus.cc:374] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:27.488054 2728 raft_consensus.cc:729] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ce64b918d6429d949afe2831bacd99, State: Initialized, Role: FOLLOWER
I20260810 03:22:27.488965 2728 consensus_queue.cc:260] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.489615 2704 heartbeater.cc:503] Master 127.19.246.254:35645 was elected leader, sending a full tablet report...
I20260810 03:22:27.490473 2726 ts_tablet_manager.cc:1428] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Time spent starting tablet: real 0.031s user 0.013s sys 0.016s
I20260810 03:22:27.488544 2729 raft_consensus.cc:348] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.493661 2735 raft_consensus.cc:482] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:22:27.494045 2729 raft_consensus.cc:374] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:27.494295 2729 raft_consensus.cc:729] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13faa5977b464174a6a30d87405505ca, State: Initialized, Role: FOLLOWER
I20260810 03:22:27.493959 2735 raft_consensus.cc:504] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.495182 2729 consensus_queue.cc:260] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.507974 2728 ts_tablet_manager.cc:1428] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: Time spent starting tablet: real 0.032s user 0.017s sys 0.003s
I20260810 03:22:27.508471 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.498680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:34509 (local address 127.19.246.195:41507)
0810 03:22:27.498954 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:27.498986 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:27.499172 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:27.499990 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:27.499996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:27.500026 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:27.500371 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.500388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.501840 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.501845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.504123 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.504151 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.507184 (+ 3033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.507195 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.507585 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.507661 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 03:22:27.507804 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 03:22:27.508043 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
W20260810 03:22:27.509346 2705 tablet.cc:2369] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:22:27.509979 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.499215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:41507 (local address 127.19.246.193:34509)
0810 03:22:27.499495 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.499501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.499525 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:22:27.499615 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.499621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.499699 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.499803 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:27.500575 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.501638 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.504550 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.506628 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.506901 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.507021 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.509334 (+ 2313us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.509578 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:22:27.511132 2341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdff41f5c47848af8ae5de7518d714fb" candidate_uuid: "c33882cb5bb04435973310f770025c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ce64b918d6429d949afe2831bacd99" is_pre_election: true
I20260810 03:22:27.512408 2341 raft_consensus.cc:2454] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33882cb5bb04435973310f770025c22 in term 0.
I20260810 03:22:27.514052 2576 leader_election.cc:302] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ce64b918d6429d949afe2831bacd99, c33882cb5bb04435973310f770025c22; no voters:
I20260810 03:22:27.514938 2733 raft_consensus.cc:2781] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:22:27.515162 2733 raft_consensus.cc:482] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:22:27.515332 2733 raft_consensus.cc:3037] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:27.515612 2735 leader_election.cc:288] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ce64b918d6429d949afe2831bacd99 (127.19.246.193:34509), 13faa5977b464174a6a30d87405505ca (127.19.246.194:34987)
I20260810 03:22:27.517488 2729 ts_tablet_manager.cc:1428] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Time spent starting tablet: real 0.046s user 0.022s sys 0.015s
I20260810 03:22:27.519588 2733 raft_consensus.cc:504] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.521525 2341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdff41f5c47848af8ae5de7518d714fb" candidate_uuid: "c33882cb5bb04435973310f770025c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ce64b918d6429d949afe2831bacd99"
I20260810 03:22:27.521871 2341 raft_consensus.cc:3037] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:27.524355 2733 leader_election.cc:288] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [CANDIDATE]: Term 1 election: Requested vote from peers 49ce64b918d6429d949afe2831bacd99 (127.19.246.193:34509), 13faa5977b464174a6a30d87405505ca (127.19.246.194:34987)
I20260810 03:22:27.525939 2341 raft_consensus.cc:2454] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33882cb5bb04435973310f770025c22 in term 1.
I20260810 03:22:27.526618 2576 leader_election.cc:302] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ce64b918d6429d949afe2831bacd99, c33882cb5bb04435973310f770025c22; no voters:
I20260810 03:22:27.526712 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.518968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:58283 (local address 127.19.246.194:34987)
0810 03:22:27.519429 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.519437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.519470 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:22:27.519541 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.519547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.519627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.519792 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:27.520565 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.521779 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.524254 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.525570 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.525772 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.525860 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.526303 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.526463 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260810 03:22:27.526729 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.515178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:34987 (local address 127.19.246.195:58283)
0810 03:22:27.518807 (+ 3629us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:27.518833 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:27.518980 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:27.519999 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:27.520004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:27.520034 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:27.520372 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.520388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.522025 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.522031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.524017 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:27.524041 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.525786 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.525793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:27.525983 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.526086 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 03:22:27.526236 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 03:22:27.526393 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3476}
I20260810 03:22:27.527099 2733 raft_consensus.cc:2781] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:22:27.527294 2733 raft_consensus.cc:686] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 1 LEADER]: Becoming Leader. State: Replica: c33882cb5bb04435973310f770025c22, State: Running, Role: LEADER
I20260810 03:22:27.527920 2733 consensus_queue.cc:237] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } }
I20260810 03:22:27.527971 2499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdff41f5c47848af8ae5de7518d714fb" candidate_uuid: "c33882cb5bb04435973310f770025c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13faa5977b464174a6a30d87405505ca" is_pre_election: true
I20260810 03:22:27.528532 2499 raft_consensus.cc:2454] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33882cb5bb04435973310f770025c22 in term 0.
I20260810 03:22:27.529341 2498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdff41f5c47848af8ae5de7518d714fb" candidate_uuid: "c33882cb5bb04435973310f770025c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13faa5977b464174a6a30d87405505ca"
I20260810 03:22:27.529630 2498 raft_consensus.cc:3037] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:27.533154 2498 raft_consensus.cc:2454] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33882cb5bb04435973310f770025c22 in term 1.
I20260810 03:22:27.533638 2175 catalog_manager.cc:5495] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33882cb5bb04435973310f770025c22 (127.19.246.195). New cstate: current_term: 1 leader_uuid: "c33882cb5bb04435973310f770025c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33882cb5bb04435973310f770025c22" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 } health_report { overall_health: UNKNOWN } } }
I20260810 03:22:27.587423 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.581297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.19.246.195:45397)
0810 03:22:27.581506 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.581512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.581625 (+ 113us) server_negotiation.cc:407] Connection header received
0810 03:22:27.581924 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.581929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.582015 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.582197 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:22:27.582598 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.583413 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.585003 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.585441 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.585588 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.586184 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:22:27.586730 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:22:27.586918 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.586980 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.587073 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:22:27.597182 2341 raft_consensus.cc:1261] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 1 FOLLOWER]: Refusing update from remote peer c33882cb5bb04435973310f770025c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:22:27.597239 2498 raft_consensus.cc:1261] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 1 FOLLOWER]: Refusing update from remote peer c33882cb5bb04435973310f770025c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:22:27.598379 2733 consensus_queue.cc:1035] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13faa5977b464174a6a30d87405505ca" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:22:27.598758 2735 consensus_queue.cc:1035] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ce64b918d6429d949afe2831bacd99" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:22:27.615643 2550 tablet.cc:2369] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:22:27.619498 2740 mvcc.cc:205] Tried to move back new op lower bound from 7316816476546260992 to 7316816476279291904. Current Snapshot: MvccSnapshot[applied={T|T < 7316816476546260992}]
I20260810 03:22:27.650441 2175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:22:27.650794 2175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:22:27.655717 2175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c4e1ad618d447258974edc508e75022: Sending DeleteTablet for 3 replicas of tablet cdff41f5c47848af8ae5de7518d714fb
I20260810 03:22:27.658505 2318 tablet_service.cc:1514] Processing DeleteTablet for tablet cdff41f5c47848af8ae5de7518d714fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:27 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:42774
I20260810 03:22:27.658425 2625 tablet_service.cc:1514] Processing DeleteTablet for tablet cdff41f5c47848af8ae5de7518d714fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:27 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:35670
I20260810 03:22:27.664623 2476 tablet_service.cc:1514] Processing DeleteTablet for tablet cdff41f5c47848af8ae5de7518d714fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:27 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:57110
I20260810 03:22:27.673269 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 2214
I20260810 03:22:27.676169 2752 tablet_replica.cc:330] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: stopping tablet replica
I20260810 03:22:27.677587 2755 tablet_replica.cc:330] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: stopping tablet replica
I20260810 03:22:27.678079 2755 raft_consensus.cc:2229] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:22:27.678804 2755 raft_consensus.cc:2258] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:27.680364 2752 raft_consensus.cc:2229] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:22:27.680950 2752 raft_consensus.cc:2258] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:27.683805 2756 tablet_replica.cc:330] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: stopping tablet replica
I20260810 03:22:27.684451 2756 raft_consensus.cc:2229] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:22:27.684554 2752 ts_tablet_manager.cc:1905] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:22:27.685065 2756 raft_consensus.cc:2258] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:27.692988 2752 ts_tablet_manager.cc:1918] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:27.693229 2755 ts_tablet_manager.cc:1905] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:22:27.693331 2752 log.cc:1196] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/cdff41f5c47848af8ae5de7518d714fb
I20260810 03:22:27.694129 2752 ts_tablet_manager.cc:1939] T cdff41f5c47848af8ae5de7518d714fb P 49ce64b918d6429d949afe2831bacd99: Deleting consensus metadata
I20260810 03:22:27.694999 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.675216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.19.246.193:34509)
0810 03:22:27.675473 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.675479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.675503 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:22:27.675582 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.675588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.675694 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.675827 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:27.678024 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.679239 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.681951 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.682602 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.682754 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.686574 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:27.686592 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:27.686610 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:27.686672 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:27.691134 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.692183 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.692194 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.692197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.692388 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.692865 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.692870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.692874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.693413 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:27.693790 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.694136 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.694472 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:22:27.696189 2385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:27.696473 2159 catalog_manager.cc:4841] TS 49ce64b918d6429d949afe2831bacd99 (127.19.246.193:34509): tablet cdff41f5c47848af8ae5de7518d714fb (table test-table [id=9e8d33375c6b4aa0a0f01f4a27e5ffb1]) successfully deleted
I20260810 03:22:27.703199 2755 ts_tablet_manager.cc:1918] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:27.703464 2756 ts_tablet_manager.cc:1905] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:22:27.703855 2755 log.cc:1196] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/cdff41f5c47848af8ae5de7518d714fb
I20260810 03:22:27.704732 2755 ts_tablet_manager.cc:1939] T cdff41f5c47848af8ae5de7518d714fb P c33882cb5bb04435973310f770025c22: Deleting consensus metadata
I20260810 03:22:27.707131 2156 catalog_manager.cc:4841] TS c33882cb5bb04435973310f770025c22 (127.19.246.195:45397): tablet cdff41f5c47848af8ae5de7518d714fb (table test-table [id=9e8d33375c6b4aa0a0f01f4a27e5ffb1]) successfully deleted
I20260810 03:22:27.714162 2756 ts_tablet_manager.cc:1918] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:27.714443 2756 log.cc:1196] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/cdff41f5c47848af8ae5de7518d714fb
I20260810 03:22:27.715914 2756 ts_tablet_manager.cc:1939] T cdff41f5c47848af8ae5de7518d714fb P 13faa5977b464174a6a30d87405505ca: Deleting consensus metadata
I20260810 03:22:27.718866 2159 catalog_manager.cc:4841] TS 13faa5977b464174a6a30d87405505ca (127.19.246.194:34987): tablet cdff41f5c47848af8ae5de7518d714fb (table test-table [id=9e8d33375c6b4aa0a0f01f4a27e5ffb1]) successfully deleted
I20260810 03:22:27.815397 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 2214
I20260810 03:22:27.838209 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 2395
I20260810 03:22:27.857635 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.839365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.19.246.194:34987)
0810 03:22:27.839596 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.839601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.839628 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:22:27.839689 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.839693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.839754 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.839877 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:27.841334 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.842341 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.844130 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.844558 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.844717 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.848524 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:27.848553 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:27.848574 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:27.848629 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:27.853020 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.854097 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.854107 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.854112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.854304 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:27.854834 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:27.854839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:27.854843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:27.855389 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:27.855731 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:27.856959 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0810 03:22:27.857219 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:22:27.858801 2544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:27.965210 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 2395
I20260810 03:22:27.987973 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 2553
I20260810 03:22:28.005962 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:27.988874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.19.246.195:45397)
0810 03:22:27.989109 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:27.989114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:27.989135 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:22:27.989499 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:27.989506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:27.989635 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:27.989770 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:27.991460 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.992436 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.993939 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:27.994480 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:27.994645 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:27.998194 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:27.998210 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:27.998227 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:27.998291 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:28.002297 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:28.003344 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:28.003483 (+ 139us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:28.003486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:28.003686 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:28.004140 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:28.004145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:28.004148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:28.004550 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:28.004942 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:28.005255 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 03:22:28.005506 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:22:28.006673 2699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:28.135624 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 2553
I20260810 03:22:28.164861 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 2136
I20260810 03:22:28.166412 2207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:28.299436 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 2136
I20260810 03:22:28.334597 20443 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 03:22:28.681885 20443 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 03:22:28.682122 20443 mini_ranger.cc:66] Stopping Ranger...
I20260810 03:22:29.029275 20443 mini_ranger.cc:68] Stopped Ranger
2026-08-10 03:22:29.029 UTC [1082] LOG: received smart shutdown request
2026-08-10 03:22:29.033 UTC [1082] LOG: background worker "logical replication launcher" (PID 1092) exited with exit code 1
2026-08-10 03:22:29.034 UTC [1086] LOG: shutting down
2026-08-10 03:22:29.242 UTC [1082] LOG: database system is shut down
2026-08-10T03:22:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (62539 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:22:29 dist-test-slave-btw8 krb5kdc[2770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:22:29 dist-test-slave-btw8 krb5kdc[2770](info): set up 2 sockets
Aug 10 03:22:29 dist-test-slave-btw8 krb5kdc[2770](info): commencing operation
krb5kdc: starting...
W20260810 03:22:30.636595 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.004s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:22:30 dist-test-slave-btw8 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:22:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:22:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:22:30.964282 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:34953
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:34953,127.19.246.253:33281,127.19.246.252:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:31.344484 2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:31.344944 2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:31.345125 2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:31.353844 2784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:31.354066 2784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:31.354202 2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:31.354311 2784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:31.354413 2784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:31.365568 2784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:34953,127.19.246.253:33281,127.19.246.252:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:34953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:31.367178 2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:31.368700 2784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:31.383785 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:31.383744 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:31.383620 2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:31.385579 2784 server_base.cc:1034] running on GCE node
I20260810 03:22:31.386830 2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:31.388693 2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:31.389992 2784 hybrid_clock.cc:648] HybridClock initialized: now 1786332151389943 us; error 76 us; skew 500 ppm
I20260810 03:22:31.390434 2784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:31 dist-test-slave-btw8 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332151, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:31.394300 2784 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:22:31.396404 2784 webserver.cc:460] Webserver started at http://127.19.246.254:33895/ using document root <none> and password file <none>
I20260810 03:22:31.397207 2784 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:31.397372 2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:31.397750 2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:22:31.401276 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "f46fd8e84df7464eb0a0f77db4239f6c"
format_stamp: "Formatted at 2026-08-10 03:22:31 on dist-test-slave-btw8"
server_key: "c67a0b22769c2399702778df38efb21c"
server_key_iv: "595a1e3a0cdafb238b495a1108f6b6ab"
server_key_version: "encryptionkey@0"
I20260810 03:22:31.402230 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "f46fd8e84df7464eb0a0f77db4239f6c"
format_stamp: "Formatted at 2026-08-10 03:22:31 on dist-test-slave-btw8"
server_key: "c67a0b22769c2399702778df38efb21c"
server_key_iv: "595a1e3a0cdafb238b495a1108f6b6ab"
server_key_version: "encryptionkey@0"
I20260810 03:22:31.409092 2784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 03:22:31.414327 2799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:31.416271 2784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:22:31.416554 2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "f46fd8e84df7464eb0a0f77db4239f6c"
format_stamp: "Formatted at 2026-08-10 03:22:31 on dist-test-slave-btw8"
server_key: "c67a0b22769c2399702778df38efb21c"
server_key_iv: "595a1e3a0cdafb238b495a1108f6b6ab"
server_key_version: "encryptionkey@0"
I20260810 03:22:31.416858 2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:31.441321 2784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:31.446532 2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:31.446915 2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:31.469525 2784 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:34953
I20260810 03:22:31.469525 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:34953 every 8 connection(s)
I20260810 03:22:31.471166 2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:22:31.475853 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:31.479203 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 2784
I20260810 03:22:31.479514 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:22:31.480435 20443 external_mini_cluster.cc:1428] Setting key ec5021085cb609b35a0d52f512c59836
I20260810 03:22:31.486333 2855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:22:31 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332151, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.246.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.253@KRBTEST.COM" created.
I20260810 03:22:31.519538 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.492731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.19.246.254:34953)
0810 03:22:31.493626 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:31.493636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:31.493717 (+ 81us) server_negotiation.cc:407] Connection header received
0810 03:22:31.495639 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:31.495650 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:31.495939 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:31.496148 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:31.498134 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:31.499860 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:31.502473 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:31.503193 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:31.503511 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:31.508433 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:31.508443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:31.508459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:31.508495 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:31.513071 (+ 4576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:31.514162 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:31.514171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:31.514173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:31.514357 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:31.515090 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:31.515098 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:31.515102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:31.516123 (+ 1021us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:31.516348 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:31.517453 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0810 03:22:31.517905 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":635,"thread_start_us":436,"threads_started":1}
I20260810 03:22:31.519584 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.491087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34953 (local address 127.0.0.1:51232)
0810 03:22:31.492141 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.492210 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:31.495022 (+ 2812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:31.496246 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:31.496261 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:31.497213 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:31.497800 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:31.497831 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:31.500067 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:31.500076 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:31.502237 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:31.502263 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:31.503572 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:31.503585 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:31.503845 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:31.504749 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:31.504787 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:31.508115 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:31.513342 (+ 5227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:31.513355 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:31.513374 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:31.513983 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:31.514610 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:31.514614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:31.514617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:31.514852 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:31.516434 (+ 1582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:31.516442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:31.516735 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 03:22:31.517080 (+ 345us) client_negotiation.cc:240] Negotiation successful
0810 03:22:31.518150 (+ 1070us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":121,"thread_start_us":473,"threads_started":1}
I20260810 03:22:31.523860 2855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:22:31.525350 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.524822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49370)
0810 03:22:31.525055 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.525208 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260810 03:22:31.525803 2803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
I20260810 03:22:31.527869 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.527346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49386)
0810 03:22:31.527603 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.527682 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 03:22:31.528141 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:33281: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab.
Entry for principal kudu/127.19.246.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab.
I20260810 03:22:31.557134 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.253:33281
--webserver_interface=127.19.246.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab
--principal=kudu/127.19.246.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:34953,127.19.246.253:33281,127.19.246.252:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:22:31.577661 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } attempt: 1
I20260810 03:22:31.579046 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.578496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49388)
0810 03:22:31.578755 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.578896 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 03:22:31.580502 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.580077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49392)
0810 03:22:31.580244 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.580334 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 03:22:31.580753 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:33281: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
I20260810 03:22:31.649099 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } attempt: 2
I20260810 03:22:31.650789 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.650219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49394)
0810 03:22:31.650509 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.650615 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 03:22:31.652302 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.651853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49410)
0810 03:22:31.652072 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.652150 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 03:22:31.652647 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:33281: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
I20260810 03:22:31.744032 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } attempt: 3
I20260810 03:22:31.745688 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.745172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49420)
0810 03:22:31.745438 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.745551 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260810 03:22:31.747156 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.746672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49434)
0810 03:22:31.746890 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.746974 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260810 03:22:31.747531 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:33281: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
I20260810 03:22:31.890897 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } attempt: 4
I20260810 03:22:31.892401 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.891811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49438)
0810 03:22:31.892164 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.892259 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260810 03:22:31.893664 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:31.893278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49444)
0810 03:22:31.893433 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:31.893531 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 03:22:31.894016 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:33281: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
W20260810 03:22:31.947788 2862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:31.948148 2862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:31.948254 2862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:31.956297 2862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:31.956457 2862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:31.956512 2862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:31.956583 2862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:31.956636 2862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:31.967659 2862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:34953,127.19.246.253:33281,127.19.246.252:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab
--principal=kudu/127.19.246.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.253:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.246.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:31.969187 2862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:31.970650 2862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:31.985350 2867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:31.985337 2868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:31.985581 2870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:31.986470 2862 server_base.cc:1034] running on GCE node
I20260810 03:22:31.987514 2862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:31.989210 2862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:31.990479 2862 hybrid_clock.cc:648] HybridClock initialized: now 1786332151990410 us; error 73 us; skew 500 ppm
I20260810 03:22:31.990895 2862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:31 dist-test-slave-btw8 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332151, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:31.994783 2862 init.cc:377] Logged in from keytab as kudu/127.19.246.253@KRBTEST.COM (short username kudu)
I20260810 03:22:31.996627 2862 webserver.cc:460] Webserver started at http://127.19.246.253:35455/ using document root <none> and password file <none>
I20260810 03:22:31.997457 2862 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:31.997627 2862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:31.997970 2862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:22:32.001453 2862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/data/instance:
uuid: "8a30bdc58f254cd1aec72214754cef95"
format_stamp: "Formatted at 2026-08-10 03:22:31 on dist-test-slave-btw8"
server_key: "470b4f2bb5b99c87e806eb6d1f967ac9"
server_key_iv: "9aed9c01a39e3331f6629789b8439365"
server_key_version: "encryptionkey@0"
I20260810 03:22:32.002470 2862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/wal/instance:
uuid: "8a30bdc58f254cd1aec72214754cef95"
format_stamp: "Formatted at 2026-08-10 03:22:31 on dist-test-slave-btw8"
server_key: "470b4f2bb5b99c87e806eb6d1f967ac9"
server_key_iv: "9aed9c01a39e3331f6629789b8439365"
server_key_version: "encryptionkey@0"
I20260810 03:22:32.008780 2862 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260810 03:22:32.013167 2877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:32.014816 2862 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:22:32.015049 2862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/wal
uuid: "8a30bdc58f254cd1aec72214754cef95"
format_stamp: "Formatted at 2026-08-10 03:22:31 on dist-test-slave-btw8"
server_key: "470b4f2bb5b99c87e806eb6d1f967ac9"
server_key_iv: "9aed9c01a39e3331f6629789b8439365"
server_key_version: "encryptionkey@0"
I20260810 03:22:32.015332 2862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:32.031740 2862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:32.036484 2862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:32.036813 2862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:32.057452 2862 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.253:33281
I20260810 03:22:32.057492 2936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.253:33281 every 8 connection(s)
I20260810 03:22:32.058984 2862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
I20260810 03:22:32.061666 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 2862
I20260810 03:22:32.061923 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-1/wal/instance
I20260810 03:22:32.062433 20443 external_mini_cluster.cc:1428] Setting key 6d2165019f93b6adc22cc14735bc50e3
I20260810 03:22:32.063599 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:32.071228 2937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332151, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.246.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.252@KRBTEST.COM" created.
I20260810 03:22:32.095468 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.074623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.19.246.254:34953)
0810 03:22:32.075210 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.075215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.076303 (+ 1088us) server_negotiation.cc:407] Connection header received
0810 03:22:32.078494 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.078499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.078610 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.078747 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.080650 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.081462 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.083457 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.083953 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.084120 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.088239 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.088245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.088248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.088292 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.091778 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.092794 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.092798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.092801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.092933 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.093414 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.093418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.093420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.093786 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.093958 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.094785 (+ 827us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.095029 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":330,"threads_started":1}
I20260810 03:22:32.096560 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.075213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34953 (local address 127.0.0.1:51234)
0810 03:22:32.076113 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.076162 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:32.078154 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:32.079040 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:32.079050 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:32.079736 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:32.080401 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.080422 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.081832 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.081837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.083228 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.083243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.084244 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.084252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.084509 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.085367 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:32.085398 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:32.087922 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:32.091997 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.092006 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.092024 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.092574 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.093060 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.093067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.093075 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.093272 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.093937 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:32.093946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:32.094387 (+ 441us) client_negotiation.cc:769] Sending connection context
0810 03:22:32.094672 (+ 285us) client_negotiation.cc:240] Negotiation successful
0810 03:22:32.095251 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":372,"threads_started":1}
I20260810 03:22:32.098649 2937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332151, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:22:32.117686 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.099572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49456)
0810 03:22:32.099814 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.099840 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:32.099980 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:32.101305 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:32.101312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:32.101873 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:32.102086 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.102100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.104448 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.104455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.105848 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.105866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.107026 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.107033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.107304 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.108061 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:32.108081 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:32.110723 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:32.114613 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.114619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.114624 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.115060 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.115587 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.115595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.115600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.115768 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.116496 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:32.116500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:32.116685 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 03:22:32.116851 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 03:22:32.117195 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 03:22:32.117823 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.099864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.19.246.253:33281)
0810 03:22:32.100699 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.100711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.100748 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:22:32.100819 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.100831 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.101101 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.101301 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.102289 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.104277 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.106039 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.106824 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.106999 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.110988 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.111005 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.111022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.111055 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.114427 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.115225 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.115236 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.115241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.115462 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.115902 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.115907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.115911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.116339 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.116582 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.117011 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.117337 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":618,"thread_start_us":446,"threads_started":1}
I20260810 03:22:32.120652 2937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:22:32.122473 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.121848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37132)
0810 03:22:32.122150 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.122289 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 03:22:32.122884 2879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.124817 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.124407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37144)
0810 03:22:32.124585 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.124689 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 03:22:32.125288 2937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab.
Entry for principal kudu/127.19.246.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab.
I20260810 03:22:32.136260 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.252:35573
--webserver_interface=127.19.246.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab
--principal=kudu/127.19.246.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:34953,127.19.246.253:33281,127.19.246.252:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:22:32.174660 2937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 1
I20260810 03:22:32.176115 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.175603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37160)
0810 03:22:32.175828 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.175941 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 03:22:32.177295 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.176836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37172)
0810 03:22:32.177020 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.177117 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 03:22:32.177783 2937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.193321 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } attempt: 5
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332151, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:22:32.208811 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.194260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.19.246.253:33281)
0810 03:22:32.194480 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.194486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.194508 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:22:32.194639 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.194644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.194758 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.194873 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.195801 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.196497 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.197763 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.198278 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.198425 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.202229 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.202235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.202238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.202275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.205581 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.206487 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.206494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.206498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.206669 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.207117 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.207123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.207127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.207539 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.207716 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.208176 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.208436 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 03:22:32.208834 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.194129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49460)
0810 03:22:32.194337 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.194361 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:32.194495 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:32.194913 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:32.194918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:32.195424 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:32.195615 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.195633 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.196647 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.196655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.197606 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.197625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.198455 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.198461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.198650 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.199324 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:32.199343 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:32.201967 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:32.205825 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.205832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.205845 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.206322 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.206811 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.206817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.206821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.206984 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.207684 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:32.207690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:32.207864 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 03:22:32.208039 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 03:22:32.208431 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 03:22:32.210210 2855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:22:32.211563 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.211007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37182)
0810 03:22:32.211230 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.211337 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 03:22:32.212603 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.212233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37196)
0810 03:22:32.212393 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.212486 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 03:22:32.212831 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.246070 2937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 2
I20260810 03:22:32.247552 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.246875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37206)
0810 03:22:32.247150 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.247288 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 03:22:32.248718 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.248289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37208)
0810 03:22:32.248471 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.248582 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 03:22:32.248980 2937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.264075 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 1
I20260810 03:22:32.265359 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.264841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37218)
0810 03:22:32.265066 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.265185 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 03:22:32.266649 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.266255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37230)
0810 03:22:32.266448 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.266527 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260810 03:22:32.266968 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.335271 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 2
I20260810 03:22:32.336752 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.336165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37246)
0810 03:22:32.336459 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.336570 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260810 03:22:32.338030 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.337639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37262)
0810 03:22:32.337805 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.337881 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 03:22:32.338296 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.340227 2937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 3
I20260810 03:22:32.341609 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.340941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37264)
0810 03:22:32.341219 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.341362 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 03:22:32.343099 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.342480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37272)
0810 03:22:32.342768 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.342891 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260810 03:22:32.343431 2937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.444625 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 3
I20260810 03:22:32.445956 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.445448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37274)
0810 03:22:32.445685 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.445782 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 03:22:32.447124 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.446704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37280)
0810 03:22:32.446915 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.447004 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260810 03:22:32.447332 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.486687 2937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 4
I20260810 03:22:32.488268 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.487525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37292)
0810 03:22:32.487859 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.488000 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 03:22:32.489639 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.489172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37302)
0810 03:22:32.489365 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.489473 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 03:22:32.489959 2937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
W20260810 03:22:32.529578 2946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:32.529919 2946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:32.530071 2946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:32.537832 2946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:32.537964 2946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:32.538053 2946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:32.538143 2946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:32.538221 2946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:32.548645 2946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:34953,127.19.246.253:33281,127.19.246.252:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab
--principal=kudu/127.19.246.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.252:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.246.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:32.550113 2946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:32.551499 2946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:32.564961 2953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:32.565052 2955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:32.565052 2952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:32.565914 2946 server_base.cc:1034] running on GCE node
I20260810 03:22:32.566730 2946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:32.568382 2946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:32.569617 2946 hybrid_clock.cc:648] HybridClock initialized: now 1786332152569557 us; error 60 us; skew 500 ppm
I20260810 03:22:32.570068 2946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332152, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:32.573633 2946 init.cc:377] Logged in from keytab as kudu/127.19.246.252@KRBTEST.COM (short username kudu)
I20260810 03:22:32.575498 2946 webserver.cc:460] Webserver started at http://127.19.246.252:41089/ using document root <none> and password file <none>
I20260810 03:22:32.576225 2946 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:32.576335 2946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:32.576601 2946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:22:32.580070 2946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/data/instance:
uuid: "f1a2f03840fc4a01bbfe791af6614a35"
format_stamp: "Formatted at 2026-08-10 03:22:32 on dist-test-slave-btw8"
server_key: "766d1418fb08e363ff8a49c1079136e5"
server_key_iv: "80ca91125e883373d7879010dd7116ef"
server_key_version: "encryptionkey@0"
I20260810 03:22:32.580895 2946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/wal/instance:
uuid: "f1a2f03840fc4a01bbfe791af6614a35"
format_stamp: "Formatted at 2026-08-10 03:22:32 on dist-test-slave-btw8"
server_key: "766d1418fb08e363ff8a49c1079136e5"
server_key_iv: "80ca91125e883373d7879010dd7116ef"
server_key_version: "encryptionkey@0"
I20260810 03:22:32.586724 2946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:22:32.590814 2964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:32.592605 2946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:22:32.592823 2946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/wal
uuid: "f1a2f03840fc4a01bbfe791af6614a35"
format_stamp: "Formatted at 2026-08-10 03:22:32 on dist-test-slave-btw8"
server_key: "766d1418fb08e363ff8a49c1079136e5"
server_key_iv: "80ca91125e883373d7879010dd7116ef"
server_key_version: "encryptionkey@0"
I20260810 03:22:32.593065 2946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:32.617945 2946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:32.623551 2946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:32.623881 2946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:32.624634 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 4
I20260810 03:22:32.626026 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.625407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37314)
0810 03:22:32.625705 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.625842 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 03:22:32.627316 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.626874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37318)
0810 03:22:32.627066 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.627167 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260810 03:22:32.627616 2855 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:35573: Network error: Client connection negotiation failed: client connection to 127.19.246.252:35573: connect: Connection refused (error 111)
I20260810 03:22:32.645823 2946 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.252:35573
I20260810 03:22:32.645860 3017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.252:35573 every 8 connection(s)
I20260810 03:22:32.647312 2946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
I20260810 03:22:32.650288 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 2946
I20260810 03:22:32.650506 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786331905428759-20443-0/minicluster-data/master-2/wal/instance
I20260810 03:22:32.651060 20443 external_mini_cluster.cc:1428] Setting key 5c473e32d122c949d5a063eb2dbb1ccf
I20260810 03:22:32.652112 3018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:32.663043 3018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332152, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:32.682561 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.653477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.19.246.252:35573)
0810 03:22:32.654469 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.654483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.654532 (+ 49us) server_negotiation.cc:407] Connection header received
0810 03:22:32.657093 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.657122 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.657622 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.658020 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.660246 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.661691 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.664396 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.665348 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.665535 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.671839 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.671855 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.671874 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.671951 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.676722 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.677772 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.677784 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.677796 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.678103 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.678735 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.678743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.678750 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.679738 (+ 988us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.680087 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.680721 (+ 634us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.681228 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":481,"threads_started":1}
I20260810 03:22:32.687852 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.667023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:34953 (local address 127.0.0.1:51242)
0810 03:22:32.668211 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.668274 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:32.668483 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:32.669011 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:32.669029 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:32.669913 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:32.670231 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.670241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.671529 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.671541 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.673591 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.673609 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.674694 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.674707 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.674919 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.675658 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:32.675682 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:32.678677 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:32.683743 (+ 5066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.683753 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.683759 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.684343 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.684891 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.684896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.684898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.685046 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.686299 (+ 1253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:32.686311 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:32.686648 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 03:22:32.686925 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 03:22:32.687311 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":621,"threads_started":1}
I20260810 03:22:32.687860 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.667061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.19.246.254:34953)
0810 03:22:32.667730 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.667737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.668395 (+ 658us) server_negotiation.cc:407] Connection header received
0810 03:22:32.668684 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.668690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.668773 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.668889 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.670421 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.671284 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.673785 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.674437 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.674552 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.678987 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.678994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.678999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.679053 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.683491 (+ 4438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.684510 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.684520 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.684524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.684729 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.685185 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.685190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.685194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.686086 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.686274 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.687035 (+ 761us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.687311 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":323,"threads_started":1}
I20260810 03:22:32.689599 3018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332152, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:32.705999 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.690443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49462)
0810 03:22:32.690681 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.690698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:32.690779 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:32.691130 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:32.691135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:32.691705 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:32.691898 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.691907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.692871 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.692876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.693771 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.693782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.694515 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.694521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.694764 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.695276 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:32.695293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:32.698064 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:32.702380 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.702387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.702392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.703063 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.703528 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.703531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.703534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.703756 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.704717 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:32.704722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:32.704962 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 03:22:32.705187 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 03:22:32.705586 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 03:22:32.706168 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.690533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.19.246.253:33281)
0810 03:22:32.690728 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.690734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.690756 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:22:32.690913 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.690927 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.690991 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.691108 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.692049 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.692728 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.693921 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.694347 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.694483 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.698403 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.698413 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.698422 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.698476 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.702175 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.703205 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.703210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.703213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.703400 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.703951 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.703956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.703959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.704408 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.704666 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.705325 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.705735 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:22:32.706502 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.686906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.19.246.254:34953)
0810 03:22:32.688768 (+ 1862us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.688774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.688792 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:22:32.688992 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.688997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.689108 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.689219 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.691210 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.692214 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.693571 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.693990 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.694135 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.699224 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.699231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.699235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.699315 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.702745 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.703857 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.703865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.703873 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.704033 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.704714 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.704720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.704724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.705133 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.705467 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.705834 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.706100 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1757,"thread_start_us":4601,"threads_started":1}
I20260810 03:22:32.707921 3018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332152, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:22:32.724938 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.709054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37350)
0810 03:22:32.709354 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.709373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:32.709498 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:32.709856 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:32.709861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:32.710459 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:32.710686 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.710704 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.711919 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.711923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.713094 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.713112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.713904 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.713913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.714172 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.715040 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:32.715068 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:32.718058 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:32.722266 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.722271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.722276 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.722660 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.723079 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.723085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.723088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.723226 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.723845 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:32.723849 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:32.724003 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 03:22:32.724163 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 03:22:32.724506 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260810 03:22:32.725030 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.709163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.19.246.252:35573)
0810 03:22:32.709331 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.709338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.709462 (+ 124us) server_negotiation.cc:407] Connection header received
0810 03:22:32.709632 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.709637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.709715 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.709806 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.710870 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.711770 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.713265 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.713710 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.713813 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.718327 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.718335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.718341 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.718390 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.722077 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.722786 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.722793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.722796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.722968 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.723327 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.723336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.723339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.723704 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.723885 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.724250 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.724706 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":61}
I20260810 03:22:32.727973 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.709076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.19.246.253:33281)
0810 03:22:32.709287 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.709293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.709318 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:22:32.709516 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.709521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.709629 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.709731 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.711764 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.712547 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.714261 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.714933 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.715032 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.721315 (+ 6283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.721321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.721324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.721364 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.724280 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.725207 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.725216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.725222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.725471 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.725990 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.725994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.725997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.726397 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.726704 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.727205 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.727566 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:22:32.731856 3018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35: Bootstrap starting.
I20260810 03:22:32.734416 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:22:32.735977 3018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:32.737140 3018 log.cc:824] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:32.740548 3018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35: No bootstrap required, opened a new log
I20260810 03:22:32.754765 3018 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.755285 3018 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:32.755451 3018 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a2f03840fc4a01bbfe791af6614a35, State: Initialized, Role: FOLLOWER
I20260810 03:22:32.756390 3018 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.763715 3035 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:32.764249 3035 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:22:32.766580 3018 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:32.772435 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.741694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.19.246.254:34953)
0810 03:22:32.741914 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.741921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.748591 (+ 6670us) server_negotiation.cc:407] Connection header received
0810 03:22:32.749193 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.749199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.749309 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.749390 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.752296 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.753477 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.756302 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.757018 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.757169 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.763144 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.763154 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.763162 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.763223 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.767393 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.768751 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.768761 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.768767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.768988 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.769588 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.769596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.769602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.770139 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.770469 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.771310 (+ 841us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.771733 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:22:32.776279 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.741325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.19.246.252:35573)
0810 03:22:32.745103 (+ 3778us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.745115 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.751879 (+ 6764us) server_negotiation.cc:407] Connection header received
0810 03:22:32.752637 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.752648 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.752810 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.752928 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.756998 (+ 4070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.758564 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.762373 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.763225 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.763346 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.767504 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.767514 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.767521 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.767570 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.772130 (+ 4560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.773214 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.773224 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.773230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.773551 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.774157 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.774169 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.774174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.774621 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.774904 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.775555 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.775814 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3460}
I20260810 03:22:32.783336 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.740639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.19.246.253:33281)
0810 03:22:32.740841 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.740847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.762470 (+ 21623us) server_negotiation.cc:407] Connection header received
0810 03:22:32.762804 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.762813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.762914 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.763001 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.765658 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.767036 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.769610 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.770494 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.770636 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.775239 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.775247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.775252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.775307 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.779402 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.780349 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.780357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.780361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.780595 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.781209 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.781216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.781219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.781715 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.782006 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.782602 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.782901 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 03:22:32.787418 3048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:22:32.787628 3048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:22:32.789252 2937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 5
I20260810 03:22:32.784050 2946 master_runner.cc:186] Error getting master registration for 127.19.246.254:34953: OK, instance_id { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" instance_seqno: 1786332151462426 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:22:32.796398 2946 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332151, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
I20260810 03:22:32.809159 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.790008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37372)
0810 03:22:32.790463 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.790502 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:32.790637 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:32.791750 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:32.791756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:32.792343 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:32.792602 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.792620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.794626 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.794634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.796177 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.796202 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.797681 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.797689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.798007 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.798953 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:32.798983 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:32.801884 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:32.806252 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.806257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.806260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.806721 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.807239 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.807243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.807246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.807482 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.808273 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:32.808277 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:32.808474 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 03:22:32.808612 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 03:22:32.808883 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260810 03:22:32.809567 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.790873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.19.246.252:35573)
0810 03:22:32.791084 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.791093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.791121 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:22:32.791185 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.791198 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.791326 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.791493 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.792829 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.794416 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.796419 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.797251 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.797407 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.802176 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.802185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.802191 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.802241 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.806056 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.806883 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.806889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.806893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.807100 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.807612 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.807619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.807622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.808088 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.808281 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.808810 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.809090 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:22:32.815197 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95: Bootstrap starting.
I20260810 03:22:32.818815 2937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:32.820209 2937 log.cc:824] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:32.824549 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95: No bootstrap required, opened a new log
I20260810 03:22:32.832283 2937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.832710 2937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:32.832862 2937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a30bdc58f254cd1aec72214754cef95, State: Initialized, Role: FOLLOWER
I20260810 03:22:32.833479 2937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.836311 3051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:32.836867 3051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:22:32.837821 2937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:32.845660 3035 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:22:32.846050 3035 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.847941 3035 leader_election.cc:288] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd8e84df7464eb0a0f77db4239f6c (127.19.246.254:34953), 8a30bdc58f254cd1aec72214754cef95 (127.19.246.253:33281)
I20260810 03:22:32.849792 2908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1a2f03840fc4a01bbfe791af6614a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a30bdc58f254cd1aec72214754cef95" is_pre_election: true
I20260810 03:22:32.850464 2908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a2f03840fc4a01bbfe791af6614a35 in term 0.
I20260810 03:22:32.851311 2969 leader_election.cc:302] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a30bdc58f254cd1aec72214754cef95, f1a2f03840fc4a01bbfe791af6614a35; no voters:
I20260810 03:22:32.851971 3035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:22:32.852146 3035 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:22:32.852296 3035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:32.858114 3035 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.861107 2908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1a2f03840fc4a01bbfe791af6614a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a30bdc58f254cd1aec72214754cef95"
I20260810 03:22:32.861450 2908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:32.851115 2862 master_runner.cc:186] Error getting master registration for 127.19.246.254:34953: OK, instance_id { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" instance_seqno: 1786332151462426 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:22:32.866156 3035 leader_election.cc:288] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [CANDIDATE]: Term 1 election: Requested vote from peers f46fd8e84df7464eb0a0f77db4239f6c (127.19.246.254:34953), 8a30bdc58f254cd1aec72214754cef95 (127.19.246.253:33281)
I20260810 03:22:32.866986 2908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a2f03840fc4a01bbfe791af6614a35 in term 1.
I20260810 03:22:32.868700 2862 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 03:22:32.869303 3063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:22:32.869504 3063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:22:32.869982 2969 leader_election.cc:302] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a30bdc58f254cd1aec72214754cef95, f1a2f03840fc4a01bbfe791af6614a35; no voters:
I20260810 03:22:32.870414 3035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 03:22:32.876780 2971 tablet.cc:2369] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:22:32.877488 3035 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [term 1 LEADER]: Becoming Leader. State: Replica: f1a2f03840fc4a01bbfe791af6614a35, State: Running, Role: LEADER
I20260810 03:22:32.878258 3035 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.882697 3066 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a2f03840fc4a01bbfe791af6614a35. Latest consensus state: current_term: 1 leader_uuid: "f1a2f03840fc4a01bbfe791af6614a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:32.883242 3066 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:32.887308 3068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:22:32.889845 3068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:22:32.904862 2855 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } attempt: 5
I20260810 03:22:32.908990 2908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [term 1 FOLLOWER]: Refusing update from remote peer f1a2f03840fc4a01bbfe791af6614a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:22:32.911310 3066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 10 03:22:32 dist-test-slave-btw8 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332151, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
I20260810 03:22:32.927332 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.905644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:35573 (local address 127.0.0.1:37388)
0810 03:22:32.906106 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:32.906129 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:32.906256 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:32.907044 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:32.907051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:32.907688 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:32.907938 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.907952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.909965 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.909972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.911763 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:32.911784 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.913157 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.913164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:32.913359 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.914221 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:32.914247 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:32.918766 (+ 4519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:32.923931 (+ 5165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.923938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.923942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.924554 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.925111 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:32.925116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:32.925120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:32.925298 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:32.926111 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:32.926118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:32.926307 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 03:22:32.926460 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 03:22:32.926807 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316}
I20260810 03:22:32.928658 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:32.906479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.19.246.252:35573)
0810 03:22:32.906671 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:32.906676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:32.906705 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:22:32.906764 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:32.906771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:32.906849 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:32.906953 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:32.908505 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.909699 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.911983 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:32.912742 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:32.912894 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:32.919026 (+ 6132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:32.919038 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:32.919043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:32.919090 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:32.923675 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.924733 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.924740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.924744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.924915 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:32.925446 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:32.925454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:32.925463 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:32.925847 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:32.926038 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:32.927897 (+ 1859us) server_negotiation.cc:299] Negotiation successful
0810 03:22:32.928203 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:22:32.935897 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c: Bootstrap starting.
I20260810 03:22:32.940953 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:32.942619 2855 log.cc:824] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:32.952731 3051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a2f03840fc4a01bbfe791af6614a35. Latest consensus state: current_term: 1 leader_uuid: "f1a2f03840fc4a01bbfe791af6614a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:32.953130 3051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:22:32.953339 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c: No bootstrap required, opened a new log
I20260810 03:22:32.957024 3070 mvcc.cc:205] Tried to move back new op lower bound from 7316816498292408320 to 7316816498196324352. Current Snapshot: MvccSnapshot[applied={T|T < 7316816498292408320}]
I20260810 03:22:32.962838 3066 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1a2f03840fc4a01bbfe791af6614a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:32.963246 3066 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:32.966871 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.967549 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:32.967826 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd8e84df7464eb0a0f77db4239f6c, State: Initialized, Role: FOLLOWER
I20260810 03:22:32.968886 3051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1a2f03840fc4a01bbfe791af6614a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:32.969214 3051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:22:32.968917 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } }
I20260810 03:22:32.970045 3068 catalog_manager.cc:1347] Generated new cluster ID: 61b909ee6ff74d2889108401406194c3
I20260810 03:22:32.970153 3068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:22:32.981042 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:32.980811 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1a2f03840fc4a01bbfe791af6614a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd8e84df7464eb0a0f77db4239f6c"
I20260810 03:22:32.983741 2826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:32.981369 2828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1a2f03840fc4a01bbfe791af6614a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" is_pre_election: true
I20260810 03:22:32.989980 2826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [term 1 FOLLOWER]: Refusing update from remote peer f1a2f03840fc4a01bbfe791af6614a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:22:32.991586 3035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:22:32.977761 3075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:32.992313 3075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:22:33.007586 3068 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:22:33.014850 3068 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:22:33.035910 3066 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1a2f03840fc4a01bbfe791af6614a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:33.036283 3066 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:33.036635 3075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a2f03840fc4a01bbfe791af6614a35. Latest consensus state: current_term: 1 leader_uuid: "f1a2f03840fc4a01bbfe791af6614a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:33.036943 3075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:22:33.035843 3083 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1a2f03840fc4a01bbfe791af6614a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd8e84df7464eb0a0f77db4239f6c" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 34953 } } peers { permanent_uuid: "8a30bdc58f254cd1aec72214754cef95" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 33281 } } peers { permanent_uuid: "f1a2f03840fc4a01bbfe791af6614a35" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 35573 } } }
I20260810 03:22:33.037310 3083 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:22:33.056087 3068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1a2f03840fc4a01bbfe791af6614a35: Generated new TSK 0
I20260810 03:22:33.057487 3068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:22:33.078756 3091 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:22:33.871833 3063 catalog_manager.cc:1259] Loaded cluster ID: 61b909ee6ff74d2889108401406194c3
I20260810 03:22:33.871945 3063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95: loading cluster ID for follower catalog manager: success
I20260810 03:22:33.874541 3063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95: acquiring CA information for follower catalog manager: success
I20260810 03:22:33.876621 3063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a30bdc58f254cd1aec72214754cef95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:22:34.084071 3091 catalog_manager.cc:1259] Loaded cluster ID: 61b909ee6ff74d2889108401406194c3
I20260810 03:22:34.084259 3091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c: loading cluster ID for follower catalog manager: success
I20260810 03:22:34.087445 3091 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c: acquiring CA information for follower catalog manager: success
I20260810 03:22:34.089761 3091 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f46fd8e84df7464eb0a0f77db4239f6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:22:34.179857 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 2784
I20260810 03:22:34.181322 2850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:34.291005 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 2784
I20260810 03:22:34.310992 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 2862
I20260810 03:22:34.312364 2932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:34.429848 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 2862
I20260810 03:22:34.454128 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 2946
I20260810 03:22:34.455479 3013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:34.570909 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:34.570015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:33281 (local address 127.0.0.1:49496)
0810 03:22:34.570552 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:34.570706 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":307,"threads_started":1}
I20260810 03:22:34.571065 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 2946
2026-08-10T03:22:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5198 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:22:34 dist-test-slave-btw8 krb5kdc[3102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:22:34 dist-test-slave-btw8 krb5kdc[3102](info): set up 2 sockets
Aug 10 03:22:34 dist-test-slave-btw8 krb5kdc[3102](info): commencing operation
krb5kdc: starting...
W20260810 03:22:35.813498 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.003s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:22:36 dist-test-slave-btw8 krb5kdc[3102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:22:36.148386 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40985
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:40985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:36.524930 3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:36.525269 3118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:36.525421 3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:36.533507 3118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:36.533664 3118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:36.533766 3118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:36.533888 3118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:36.533964 3118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:36.544728 3118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:40985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:36.546241 3118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:36.547689 3118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:36.561074 3129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:36.561029 3127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:36.561206 3126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:36.562328 3118 server_base.cc:1034] running on GCE node
I20260810 03:22:36.563225 3118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:36.564941 3118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:36.566208 3118 hybrid_clock.cc:648] HybridClock initialized: now 1786332156566158 us; error 40 us; skew 500 ppm
I20260810 03:22:36.566766 3118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:36 dist-test-slave-btw8 krb5kdc[3102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332156, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:36.570789 3118 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:22:36.572813 3118 webserver.cc:460] Webserver started at http://127.19.246.254:42923/ using document root <none> and password file <none>
I20260810 03:22:36.573593 3118 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:36.573714 3118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:36.574020 3118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:22:36.577567 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "3967017a502c4b0b9cad535e4c60bcc1"
format_stamp: "Formatted at 2026-08-10 03:22:36 on dist-test-slave-btw8"
server_key: "6aefe55597e1ffcb2d31286e8a2cf82c"
server_key_iv: "862bfa89719ceb51cb2731ed899fa471"
server_key_version: "encryptionkey@0"
I20260810 03:22:36.578464 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "3967017a502c4b0b9cad535e4c60bcc1"
format_stamp: "Formatted at 2026-08-10 03:22:36 on dist-test-slave-btw8"
server_key: "6aefe55597e1ffcb2d31286e8a2cf82c"
server_key_iv: "862bfa89719ceb51cb2731ed899fa471"
server_key_version: "encryptionkey@0"
I20260810 03:22:36.584654 3118 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 03:22:36.588729 3136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:36.590457 3118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:22:36.590705 3118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "3967017a502c4b0b9cad535e4c60bcc1"
format_stamp: "Formatted at 2026-08-10 03:22:36 on dist-test-slave-btw8"
server_key: "6aefe55597e1ffcb2d31286e8a2cf82c"
server_key_iv: "862bfa89719ceb51cb2731ed899fa471"
server_key_version: "encryptionkey@0"
I20260810 03:22:36.590948 3118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:36.621954 3118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:36.627756 3118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:36.628163 3118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:36.651172 3118 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:40985
I20260810 03:22:36.651196 3190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:40985 every 8 connection(s)
I20260810 03:22:36.652796 3118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:22:36.653412 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 3118
I20260810 03:22:36.653663 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:22:36.654577 20443 external_mini_cluster.cc:1428] Setting key 40c5cf7fbdcbd5e1071b0244a006d206
I20260810 03:22:36.657375 3191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:36.673935 3191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1: Bootstrap starting.
Aug 10 03:22:36 dist-test-slave-btw8 krb5kdc[3102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:36.677577 3191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:36.678785 3191 log.cc:824] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:36.682111 3191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1: No bootstrap required, opened a new log
I20260810 03:22:36.684070 3194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:36.657001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.19.246.254:40985)
0810 03:22:36.660540 (+ 3539us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:36.660553 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:36.660601 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:22:36.662300 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:36.662325 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:36.662775 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:36.663130 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:36.665426 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:36.666901 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:36.669101 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:36.670021 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:36.670190 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:36.675809 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:36.675818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:36.675843 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:36.675893 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:36.679396 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:36.680302 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:36.680310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:36.680317 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:36.680503 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:36.680950 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:36.680954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:36.680956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:36.681526 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:36.681860 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:36.682385 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 03:22:36.682869 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3280,"thread_start_us":2962,"threads_started":1}
I20260810 03:22:36.689304 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:22:36.692224 3191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3967017a502c4b0b9cad535e4c60bcc1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40985 } }
I20260810 03:22:36.692783 3191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:36.692903 3191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3967017a502c4b0b9cad535e4c60bcc1, State: Initialized, Role: FOLLOWER
I20260810 03:22:36.693558 3191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3967017a502c4b0b9cad535e4c60bcc1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40985 } }
I20260810 03:22:36.693801 3191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:22:36.693907 3191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:22:36.694039 3191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:36.697569 3191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3967017a502c4b0b9cad535e4c60bcc1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40985 } }
I20260810 03:22:36.698192 3191 leader_election.cc:302] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3967017a502c4b0b9cad535e4c60bcc1; no voters:
I20260810 03:22:36.698809 3191 leader_election.cc:288] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:22:36.698979 3197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:22:36.699805 3197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [term 1 LEADER]: Becoming Leader. State: Replica: 3967017a502c4b0b9cad535e4c60bcc1, State: Running, Role: LEADER
I20260810 03:22:36.700373 3197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3967017a502c4b0b9cad535e4c60bcc1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40985 } }
I20260810 03:22:36.701082 3191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:36.703056 3200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3967017a502c4b0b9cad535e4c60bcc1. Latest consensus state: current_term: 1 leader_uuid: "3967017a502c4b0b9cad535e4c60bcc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3967017a502c4b0b9cad535e4c60bcc1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40985 } } }
I20260810 03:22:36.707566 3200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:36.707638 3199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3967017a502c4b0b9cad535e4c60bcc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3967017a502c4b0b9cad535e4c60bcc1" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40985 } } }
I20260810 03:22:36.708086 3199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:36.709010 3206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:22:36.716488 3206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:22:36.727715 3206 catalog_manager.cc:1347] Generated new cluster ID: a30071647c114452a09e55f0f2788a4e
I20260810 03:22:36.727892 3206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:22:36.746529 3206 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:22:36.748332 3206 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:22:36.763463 3206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3967017a502c4b0b9cad535e4c60bcc1: Generated new TSK 0
I20260810 03:22:36.764513 3206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:22:36.804396 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 3118
I20260810 03:22:36.805989 3186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:36.906220 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 3118
2026-08-10T03:22:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2330 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:22:37 dist-test-slave-btw8 krb5kdc[3224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:22:37 dist-test-slave-btw8 krb5kdc[3224](info): set up 2 sockets
Aug 10 03:22:37 dist-test-slave-btw8 krb5kdc[3224](info): commencing operation
krb5kdc: starting...
W20260810 03:22:38.148780 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:22:38 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:22:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:22:38.500581 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41507
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:38.896883 3238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:38.897459 3238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:38.897706 3238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:38.906883 3238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:38.907042 3238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:38.907166 3238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:38.907266 3238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:38.907346 3238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:38.919318 3238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:38.920957 3238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:38.922399 3238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:38.936122 3247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:38.936184 3244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:38.936209 3243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:38.938263 3238 server_base.cc:1034] running on GCE node
I20260810 03:22:38.939333 3238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:38.941125 3238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:38.942421 3238 hybrid_clock.cc:648] HybridClock initialized: now 1786332158942350 us; error 65 us; skew 500 ppm
I20260810 03:22:38.942857 3238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:38 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332158, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:38.946969 3238 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:22:38.948969 3238 webserver.cc:460] Webserver started at http://127.19.246.254:43263/ using document root <none> and password file <none>
I20260810 03:22:38.949816 3238 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:38.949999 3238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:38.950372 3238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:22:38.954098 3238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "d71f82d4cc7243fe90d845cace809045"
format_stamp: "Formatted at 2026-08-10 03:22:38 on dist-test-slave-btw8"
server_key: "fc7ef2530cd0ebe52f605f3f5930987f"
server_key_iv: "cee226a676af4fa1df4115b775df8527"
server_key_version: "encryptionkey@0"
I20260810 03:22:38.955065 3238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "d71f82d4cc7243fe90d845cace809045"
format_stamp: "Formatted at 2026-08-10 03:22:38 on dist-test-slave-btw8"
server_key: "fc7ef2530cd0ebe52f605f3f5930987f"
server_key_iv: "cee226a676af4fa1df4115b775df8527"
server_key_version: "encryptionkey@0"
I20260810 03:22:38.962693 3238 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260810 03:22:38.967286 3254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:38.969472 3238 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:22:38.969795 3238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "d71f82d4cc7243fe90d845cace809045"
format_stamp: "Formatted at 2026-08-10 03:22:38 on dist-test-slave-btw8"
server_key: "fc7ef2530cd0ebe52f605f3f5930987f"
server_key_iv: "cee226a676af4fa1df4115b775df8527"
server_key_version: "encryptionkey@0"
I20260810 03:22:38.970074 3238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:38.991943 3238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:38.998153 3238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:38.998579 3238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:39.021348 3238 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:41507
I20260810 03:22:39.021401 3313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:41507 every 8 connection(s)
I20260810 03:22:39.022902 3238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:22:39.026154 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 3238
I20260810 03:22:39.026402 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:22:39.027281 20443 external_mini_cluster.cc:1428] Setting key d654d87926fac1cf054a7515731ab255
I20260810 03:22:39.027740 3314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:39.043774 3314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Bootstrap starting.
Aug 10 03:22:39 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:39.047868 3314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:39.049435 3314 log.cc:824] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:39.053426 3314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: No bootstrap required, opened a new log
I20260810 03:22:39.057134 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:39.029657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.19.246.254:41507)
0810 03:22:39.030541 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:39.030554 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:39.030599 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:22:39.033110 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:39.033147 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:39.033819 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:39.034371 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:39.036046 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:39.038149 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:39.039977 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:39.041191 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:39.041418 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:39.046829 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:39.046847 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:39.046869 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:39.046933 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:39.051293 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:39.052258 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:39.052271 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:39.052281 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:39.052518 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:39.053054 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:39.053059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:39.053062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:39.053868 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:39.054347 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:39.054944 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 03:22:39.055511 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":428,"threads_started":1}
I20260810 03:22:39.063712 3314 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:39.064154 3314 raft_consensus.cc:374] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:39.064235 3314 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71f82d4cc7243fe90d845cace809045, State: Initialized, Role: FOLLOWER
I20260810 03:22:39.064795 3314 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:39.065006 3314 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:22:39.065158 3314 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:22:39.065346 3314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:39.068078 3314 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:39.068696 3314 leader_election.cc:302] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71f82d4cc7243fe90d845cace809045; no voters:
I20260810 03:22:39.069478 3314 leader_election.cc:288] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:22:39.069593 3319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:22:39.070586 3319 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 1 LEADER]: Becoming Leader. State: Replica: d71f82d4cc7243fe90d845cace809045, State: Running, Role: LEADER
I20260810 03:22:39.071259 3319 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:39.072250 3314 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:39.075117 3320 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d71f82d4cc7243fe90d845cace809045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } } }
I20260810 03:22:39.075680 3320 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:39.076572 3321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71f82d4cc7243fe90d845cace809045. Latest consensus state: current_term: 1 leader_uuid: "d71f82d4cc7243fe90d845cace809045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } } }
I20260810 03:22:39.077019 3321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:39.077618 3325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:22:39.084568 3325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:22:39.096386 3325 catalog_manager.cc:1347] Generated new cluster ID: b8f5017771d8447f8bf1a466883f005a
I20260810 03:22:39.096568 3325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:22:39.125726 3325 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:22:39.127830 3325 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:22:39.141495 3325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Generated new TSK 0
I20260810 03:22:39.142812 3325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:22:39.231331 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41507
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:39.620446 3340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:39.620806 3340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:39.621007 3340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:39.628945 3340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:39.629089 3340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:39.629276 3340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:22:39.640307 3340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:39.641852 3340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:39.643282 3340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:39.658083 3350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:39.658120 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:39.658171 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:39.664714 3340 server_base.cc:1034] running on GCE node
I20260810 03:22:39.665545 3340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:39.666644 3340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:39.667889 3340 hybrid_clock.cc:648] HybridClock initialized: now 1786332159667829 us; error 46 us; skew 500 ppm
I20260810 03:22:39.668308 3340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:39 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:39.672199 3340 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:22:39.674055 3340 webserver.cc:460] Webserver started at http://127.19.246.193:46247/ using document root <none> and password file <none>
I20260810 03:22:39.674981 3340 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:39.675107 3340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:39.675457 3340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:22:39.678778 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "0e25dc2714b745558b860e00d5a06a73"
format_stamp: "Formatted at 2026-08-10 03:22:39 on dist-test-slave-btw8"
server_key: "7275053c1e4a0d4d5e954a5150dd46c4"
server_key_iv: "2e5b2c5f23692a201de225fa92eddda2"
server_key_version: "encryptionkey@0"
I20260810 03:22:39.679751 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "0e25dc2714b745558b860e00d5a06a73"
format_stamp: "Formatted at 2026-08-10 03:22:39 on dist-test-slave-btw8"
server_key: "7275053c1e4a0d4d5e954a5150dd46c4"
server_key_iv: "2e5b2c5f23692a201de225fa92eddda2"
server_key_version: "encryptionkey@0"
I20260810 03:22:39.685914 3340 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:22:39.689937 3357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:39.691656 3340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:22:39.691865 3340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "0e25dc2714b745558b860e00d5a06a73"
format_stamp: "Formatted at 2026-08-10 03:22:39 on dist-test-slave-btw8"
server_key: "7275053c1e4a0d4d5e954a5150dd46c4"
server_key_iv: "2e5b2c5f23692a201de225fa92eddda2"
server_key_version: "encryptionkey@0"
I20260810 03:22:39.692134 3340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:39.714591 3340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:39.719735 3340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:39.720093 3340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:39.721233 3340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:39.722808 3340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:39.722892 3340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:39.723042 3340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:39.723110 3340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:39.761545 3340 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:46359
I20260810 03:22:39.761757 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:46359 every 8 connection(s)
I20260810 03:22:39.763105 3340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:22:39.767722 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 3340
I20260810 03:22:39.767968 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:22:39.768746 20443 external_mini_cluster.cc:1428] Setting key 585f2f163460276774bf607b7af76cee
Aug 10 03:22:39 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:39.791806 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:39.766477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:39709 (local address 127.19.246.254:41507)
0810 03:22:39.767159 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:39.767167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:39.768566 (+ 1399us) server_negotiation.cc:407] Connection header received
0810 03:22:39.771027 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:39.771033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:39.771128 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:39.771298 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:39.773503 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:39.774734 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:39.776893 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:39.777636 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:39.777814 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:39.783098 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:39.783110 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:39.783118 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:39.783176 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:39.787212 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:39.788674 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:39.788680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:39.788682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:39.788834 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:39.789525 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:39.789529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:39.789531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:39.789859 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:39.790009 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:39.791154 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0810 03:22:39.791493 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":327,"threads_started":1}
I20260810 03:22:39.793474 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:39.766979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:39709)
0810 03:22:39.768046 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:39.768384 (+ 338us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:39.770590 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:39.771665 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:39.771681 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:39.772525 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:39.773307 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:39.773330 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:39.775008 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:39.775018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:39.776701 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:39.776716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:39.777942 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:39.777948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:39.778200 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:39.779051 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:39.779075 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:39.782688 (+ 3613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:39.787523 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:39.787544 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:39.787571 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:39.788455 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:39.788985 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:39.788993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:39.788999 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:39.789354 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:39.790021 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:39.790033 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:39.790613 (+ 580us) client_negotiation.cc:769] Sending connection context
0810 03:22:39.790992 (+ 379us) client_negotiation.cc:240] Negotiation successful
0810 03:22:39.791722 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":449,"threads_started":1}
I20260810 03:22:39.796437 3481 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41507
I20260810 03:22:39.796828 3481 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:39.797827 3481 heartbeater.cc:511] Master 127.19.246.254:41507 requested a full tablet report, sending...
I20260810 03:22:39.800638 3275 ts_manager.cc:194] Registered new tserver with Master: 0e25dc2714b745558b860e00d5a06a73 (127.19.246.193:46359)
I20260810 03:22:39.803156 3275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:39709
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:22:39.848099 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41507
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:40.230841 3487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:40.231168 3487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:40.231427 3487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:40.239594 3487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:40.239735 3487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:40.239919 3487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:22:40.250427 3487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:40.251962 3487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:40.253448 3487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:40.268101 3494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:40.268213 3497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:40.268077 3495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:40.269106 3487 server_base.cc:1034] running on GCE node
I20260810 03:22:40.269882 3487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:40.270987 3487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:40.272213 3487 hybrid_clock.cc:648] HybridClock initialized: now 1786332160272133 us; error 85 us; skew 500 ppm
I20260810 03:22:40.272648 3487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:40 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332160, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:40.276450 3487 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:22:40.278240 3487 webserver.cc:460] Webserver started at http://127.19.246.194:38281/ using document root <none> and password file <none>
I20260810 03:22:40.279026 3487 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:40.279176 3487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:40.279587 3487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:22:40.283008 3487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "4644b27d6e574a69ad298e4e0cab176c"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "001c441ba99ac7fe9142f4d7eea14b8b"
server_key_iv: "ab43654b4dc988b17784f4c87279e866"
server_key_version: "encryptionkey@0"
I20260810 03:22:40.284001 3487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "4644b27d6e574a69ad298e4e0cab176c"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "001c441ba99ac7fe9142f4d7eea14b8b"
server_key_iv: "ab43654b4dc988b17784f4c87279e866"
server_key_version: "encryptionkey@0"
I20260810 03:22:40.289791 3487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 03:22:40.293922 3504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:40.295626 3487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:22:40.295815 3487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "4644b27d6e574a69ad298e4e0cab176c"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "001c441ba99ac7fe9142f4d7eea14b8b"
server_key_iv: "ab43654b4dc988b17784f4c87279e866"
server_key_version: "encryptionkey@0"
I20260810 03:22:40.296031 3487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:40.314157 3487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:40.319584 3487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:40.319947 3487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:40.321128 3487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:40.322798 3487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:40.322940 3487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:40.323115 3487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:40.323195 3487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:40.361085 3487 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:41131
I20260810 03:22:40.361209 3634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:41131 every 8 connection(s)
I20260810 03:22:40.362496 3487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:22:40.363673 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 3487
I20260810 03:22:40.363889 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:22:40.364495 20443 external_mini_cluster.cc:1428] Setting key 2a366e3183b0edd4bb68defdc48b61a1
Aug 10 03:22:40 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332160, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:40.391485 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:40.365844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:50289 (local address 127.19.246.254:41507)
0810 03:22:40.366389 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:40.366399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:40.367639 (+ 1240us) server_negotiation.cc:407] Connection header received
0810 03:22:40.370802 (+ 3163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:40.370808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:40.370893 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:40.371009 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:40.373620 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.374452 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.377395 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.377851 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.377966 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:40.382900 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:40.382911 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:40.382919 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:40.382974 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:40.387329 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:40.388405 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:40.388414 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:40.388420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:40.388627 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:40.389122 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:40.389130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:40.389136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:40.389567 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:40.389746 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:40.390598 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 03:22:40.390878 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":287,"threads_started":1}
I20260810 03:22:40.392324 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:40.366486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:50289)
0810 03:22:40.367437 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:40.367489 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:40.370349 (+ 2860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:40.371394 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:40.371410 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:40.372326 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:40.373287 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:40.373320 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.374647 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.374655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:40.377181 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:40.377203 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.378126 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.378136 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:40.378492 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:40.379388 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:40.379427 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:40.382518 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:40.387666 (+ 5148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:40.387675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:40.387692 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:40.388212 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:40.388766 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:40.388769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:40.388771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:40.388984 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:40.389746 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:40.389753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:40.390159 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 03:22:40.390431 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 03:22:40.390945 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":377,"threads_started":1}
I20260810 03:22:40.394061 3635 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41507
I20260810 03:22:40.394428 3635 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:40.395401 3635 heartbeater.cc:511] Master 127.19.246.254:41507 requested a full tablet report, sending...
I20260810 03:22:40.397250 3275 ts_manager.cc:194] Registered new tserver with Master: 4644b27d6e574a69ad298e4e0cab176c (127.19.246.194:41131)
I20260810 03:22:40.398180 3275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:50289
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:22:40.440487 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41507
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:40.804989 3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:40.805385 3641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:40.805574 3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 03:22:40.807420 3481 heartbeater.cc:503] Master 127.19.246.254:41507 was elected leader, sending a full tablet report...
W20260810 03:22:40.813922 3641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:40.814132 3641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:40.814338 3641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:22:40.825078 3641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:40.826617 3641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:40.828186 3641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:40.843045 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:40.843024 3648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:40.843127 3650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:40.844195 3641 server_base.cc:1034] running on GCE node
I20260810 03:22:40.845019 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:40.846105 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:40.847448 3641 hybrid_clock.cc:648] HybridClock initialized: now 1786332160847348 us; error 60 us; skew 500 ppm
I20260810 03:22:40.847947 3641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:40 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332160, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:40.851758 3641 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:22:40.853546 3641 webserver.cc:460] Webserver started at http://127.19.246.195:40553/ using document root <none> and password file <none>
I20260810 03:22:40.854321 3641 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:40.854444 3641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:40.854830 3641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:22:40.858325 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "24a4338e83684a379c920b9fda7ebdbe"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "537f2991f06d10fa13654ce4b637607f"
server_key_iv: "0bdcea3090319cbde42f917a4e907e4c"
server_key_version: "encryptionkey@0"
I20260810 03:22:40.859309 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "24a4338e83684a379c920b9fda7ebdbe"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "537f2991f06d10fa13654ce4b637607f"
server_key_iv: "0bdcea3090319cbde42f917a4e907e4c"
server_key_version: "encryptionkey@0"
I20260810 03:22:40.865304 3641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:22:40.869248 3657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:40.871109 3641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:22:40.871450 3641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "24a4338e83684a379c920b9fda7ebdbe"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "537f2991f06d10fa13654ce4b637607f"
server_key_iv: "0bdcea3090319cbde42f917a4e907e4c"
server_key_version: "encryptionkey@0"
I20260810 03:22:40.871723 3641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:40.890894 3641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:40.896773 3641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:40.897146 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:40.898495 3641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:40.901114 3641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:40.901237 3641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:40.901425 3641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:40.901504 3641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:40.941777 3641 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:39827
I20260810 03:22:40.941838 3784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:39827 every 8 connection(s)
I20260810 03:22:40.943243 3641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:22:40.953696 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 3641
I20260810 03:22:40.953927 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:22:40.954510 20443 external_mini_cluster.cc:1428] Setting key 795503bbda473ad0394f66ce9c1d4a55
Aug 10 03:22:40 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332160, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:40.966841 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:40.946148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:43025 (local address 127.19.246.254:41507)
0810 03:22:40.946690 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:40.946697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:40.947899 (+ 1202us) server_negotiation.cc:407] Connection header received
0810 03:22:40.949944 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:40.949949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:40.950028 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:40.950142 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:40.952031 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.952901 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.954818 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.955502 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.955646 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:40.960237 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:40.960243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:40.960246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:40.960286 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:40.963301 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:40.964392 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:40.964396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:40.964399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:40.964557 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:40.965065 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:40.965068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:40.965071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:40.965386 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:40.965517 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:40.966342 (+ 825us) server_negotiation.cc:299] Negotiation successful
0810 03:22:40.966543 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":264,"threads_started":1}
I20260810 03:22:40.968058 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:40.946774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:43025)
0810 03:22:40.947705 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:40.947758 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:40.949622 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:40.950441 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:40.950451 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:40.951021 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:40.951821 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:40.951841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.953080 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.953092 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:40.954627 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:40.954645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.955786 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.955793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:40.956027 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:40.957020 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:40.957054 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:40.959945 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:40.963635 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:40.963644 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:40.963660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:40.964223 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:40.964706 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:40.964710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:40.964713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:40.964930 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:40.965522 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:40.965529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:40.965968 (+ 439us) client_negotiation.cc:769] Sending connection context
0810 03:22:40.966214 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 03:22:40.966682 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":331,"threads_started":1}
I20260810 03:22:40.970202 3785 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41507
I20260810 03:22:40.970652 3785 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:40.971946 3785 heartbeater.cc:511] Master 127.19.246.254:41507 requested a full tablet report, sending...
I20260810 03:22:40.973713 3275 ts_manager.cc:194] Registered new tserver with Master: 24a4338e83684a379c920b9fda7ebdbe (127.19.246.195:39827)
I20260810 03:22:40.974647 3275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:43025
I20260810 03:22:40.985486 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:22:41.005822 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:40.990656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.19.246.254:41507)
0810 03:22:40.990929 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:40.990937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:40.991474 (+ 537us) server_negotiation.cc:407] Connection header received
0810 03:22:40.991669 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:40.991673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:40.991756 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:40.991850 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:40.993453 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.994270 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.995864 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:40.996376 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:40.996529 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:40.999310 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:40.999316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:40.999319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:40.999423 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:41.002343 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:41.003314 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:41.003318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:41.003321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:41.003517 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:41.004244 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:41.004249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:41.004251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:41.004637 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:41.004852 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.005237 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.005505 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:22:41.011036 3275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:22:41.018621 3275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:22:41.073604 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.034711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.19.246.193:46359)
0810 03:22:41.059338 (+ 24627us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.059626 (+ 288us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.060997 (+ 1371us) server_negotiation.cc:407] Connection header received
0810 03:22:41.061161 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.061174 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.061558 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.061804 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:41.063892 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.065941 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.069135 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.070896 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.071133 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.071808 (+ 675us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.072980 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.073196 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24380,"thread_start_us":24665,"threads_started":1}
I20260810 03:22:41.074455 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.033936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:54950)
0810 03:22:41.060301 (+ 26365us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.060409 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:41.060711 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:41.062718 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:41.062731 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:41.062790 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:41.063496 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.063517 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.066130 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.066135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.068945 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.068958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.071157 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.071164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.072171 (+ 1007us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.072259 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 03:22:41.072392 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 03:22:41.073970 (+ 1578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26077,"mutex_wait_us":678,"thread_start_us":31554,"threads_started":1}
I20260810 03:22:41.074621 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.034294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.19.246.194:41131)
0810 03:22:41.057488 (+ 23194us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.057504 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.065026 (+ 7522us) server_negotiation.cc:407] Connection header received
0810 03:22:41.065416 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.065429 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.065754 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.065979 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:41.067481 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.069106 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.071923 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.073362 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.073569 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.073832 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.074090 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.074261 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22933,"thread_start_us":22726,"threads_started":1}
I20260810 03:22:41.074802 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.034005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:41131 (local address 127.0.0.1:58144)
0810 03:22:41.064873 (+ 30868us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.064896 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:41.065176 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:41.066323 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:41.066328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:41.066357 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:41.067202 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.067220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.069265 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.069274 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.071695 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.071711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.073591 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.073597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.073849 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.073892 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 03:22:41.074012 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 03:22:41.074176 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30721,"mutex_wait_us":110,"thread_start_us":30544,"threads_started":1}
I20260810 03:22:41.075912 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.034409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:39827 (local address 127.0.0.1:47122)
0810 03:22:41.065794 (+ 31385us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.065818 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:41.065950 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:41.067053 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:41.067061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:41.067099 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:41.067646 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.067660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.069928 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.069936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.072086 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.072101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.074854 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.074864 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.075092 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.075126 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:22:41.075257 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:22:41.075489 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31249,"mutex_wait_us":101,"thread_start_us":25442,"threads_started":1}
I20260810 03:22:41.075966 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.035006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.19.246.195:39827)
0810 03:22:41.064042 (+ 29036us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.064068 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.066097 (+ 2029us) server_negotiation.cc:407] Connection header received
0810 03:22:41.066219 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.066229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.066476 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.066738 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:41.067884 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.069757 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.072310 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.074260 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.074469 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.074678 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.075340 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.075543 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28766,"thread_start_us":29073,"threads_started":1}
I20260810 03:22:41.080638 3557 tablet_service.cc:1467] Processing CreateTablet for tablet 8a15205d44584a7fbb848ddfdec07588 (DEFAULT_TABLE table=test-table [id=61856bb719a74f5dbc4cf8ff8b85938b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:41.080638 3411 tablet_service.cc:1467] Processing CreateTablet for tablet 8a15205d44584a7fbb848ddfdec07588 (DEFAULT_TABLE table=test-table [id=61856bb719a74f5dbc4cf8ff8b85938b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:41.082362 3557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a15205d44584a7fbb848ddfdec07588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:41.082361 3411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a15205d44584a7fbb848ddfdec07588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:41.081825 3709 tablet_service.cc:1467] Processing CreateTablet for tablet 8a15205d44584a7fbb848ddfdec07588 (DEFAULT_TABLE table=test-table [id=61856bb719a74f5dbc4cf8ff8b85938b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:41.083575 3709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a15205d44584a7fbb848ddfdec07588. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:41.099056 3806 tablet_bootstrap.cc:492] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Bootstrap starting.
I20260810 03:22:41.101613 3808 tablet_bootstrap.cc:492] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Bootstrap starting.
I20260810 03:22:41.103725 3806 tablet_bootstrap.cc:654] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:41.103713 3807 tablet_bootstrap.cc:492] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Bootstrap starting.
I20260810 03:22:41.105253 3806 log.cc:824] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:41.105950 3808 tablet_bootstrap.cc:654] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:41.107184 3807 tablet_bootstrap.cc:654] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:41.107698 3808 log.cc:824] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:41.108592 3807 log.cc:824] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:41.109920 3806 tablet_bootstrap.cc:492] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: No bootstrap required, opened a new log
I20260810 03:22:41.110314 3806 ts_tablet_manager.cc:1397] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 03:22:41.110855 3808 tablet_bootstrap.cc:492] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: No bootstrap required, opened a new log
I20260810 03:22:41.111153 3808 ts_tablet_manager.cc:1397] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 03:22:41.111979 3807 tablet_bootstrap.cc:492] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: No bootstrap required, opened a new log
I20260810 03:22:41.112279 3807 ts_tablet_manager.cc:1397] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 03:22:41.119477 3808 raft_consensus.cc:348] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.119942 3808 raft_consensus.cc:374] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:41.120076 3808 raft_consensus.cc:729] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4644b27d6e574a69ad298e4e0cab176c, State: Initialized, Role: FOLLOWER
I20260810 03:22:41.120779 3808 consensus_queue.cc:260] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.122035 3635 heartbeater.cc:503] Master 127.19.246.254:41507 was elected leader, sending a full tablet report...
I20260810 03:22:41.121979 3806 raft_consensus.cc:348] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.122458 3806 raft_consensus.cc:374] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:41.122570 3806 raft_consensus.cc:729] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a4338e83684a379c920b9fda7ebdbe, State: Initialized, Role: FOLLOWER
I20260810 03:22:41.122948 3808 ts_tablet_manager.cc:1428] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 03:22:41.123209 3806 consensus_queue.cc:260] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.123459 3807 raft_consensus.cc:348] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.124059 3807 raft_consensus.cc:374] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:41.124223 3807 raft_consensus.cc:729] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e25dc2714b745558b860e00d5a06a73, State: Initialized, Role: FOLLOWER
I20260810 03:22:41.124461 3785 heartbeater.cc:503] Master 127.19.246.254:41507 was elected leader, sending a full tablet report...
I20260810 03:22:41.124997 3807 consensus_queue.cc:260] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.125234 3806 ts_tablet_manager.cc:1428] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260810 03:22:41.126892 3807 ts_tablet_manager.cc:1428] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Time spent starting tablet: real 0.014s user 0.003s sys 0.011s
W20260810 03:22:41.195190 3786 tablet.cc:2369] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:22:41.266665 3482 tablet.cc:2369] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:22:41.322420 3813 raft_consensus.cc:482] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:22:41.322729 3813 raft_consensus.cc:504] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.324702 3813 leader_election.cc:288] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e25dc2714b745558b860e00d5a06a73 (127.19.246.193:46359), 24a4338e83684a379c920b9fda7ebdbe (127.19.246.195:39827)
I20260810 03:22:41.336184 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.325520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:47797 (local address 127.19.246.193:46359)
0810 03:22:41.325755 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.325762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.326653 (+ 891us) server_negotiation.cc:407] Connection header received
0810 03:22:41.326835 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.326842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.326927 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.327038 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:41.327967 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.329172 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.333108 (+ 3936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.334637 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.334866 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.334952 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.335634 (+ 682us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.335866 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:22:41.336304 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.325186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:39827 (local address 127.19.246.194:43155)
0810 03:22:41.326386 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.326406 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:41.326655 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:41.327084 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:41.327089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:41.327118 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:41.327823 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.327838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.329375 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.329382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.332783 (+ 3401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.332810 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.335048 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.335056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.335377 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.335459 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 03:22:41.335571 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 03:22:41.335762 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":320,"thread_start_us":945,"threads_started":1}
I20260810 03:22:41.336329 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.325205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.19.246.194:47797)
0810 03:22:41.326525 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.326540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:41.326655 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:41.327107 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:41.327112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:41.327129 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:41.327709 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.327733 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.329375 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.329382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.332781 (+ 3399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:41.332810 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.335048 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.335056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:41.335341 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.335451 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 03:22:41.335571 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 03:22:41.335740 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":273,"thread_start_us":482,"threads_started":1}
I20260810 03:22:41.336784 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.325287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:43155 (local address 127.19.246.195:39827)
0810 03:22:41.325507 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.325513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.326548 (+ 1035us) server_negotiation.cc:407] Connection header received
0810 03:22:41.326835 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.326842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.326927 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.327038 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:41.327990 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.329159 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.333096 (+ 3937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.334637 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.335267 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.335342 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.336221 (+ 879us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.336395 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:22:41.338013 3432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a15205d44584a7fbb848ddfdec07588" candidate_uuid: "4644b27d6e574a69ad298e4e0cab176c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e25dc2714b745558b860e00d5a06a73" is_pre_election: true
I20260810 03:22:41.338548 3432 raft_consensus.cc:2454] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4644b27d6e574a69ad298e4e0cab176c in term 0.
I20260810 03:22:41.339371 3507 leader_election.cc:302] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e25dc2714b745558b860e00d5a06a73, 4644b27d6e574a69ad298e4e0cab176c; no voters:
I20260810 03:22:41.339497 3732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a15205d44584a7fbb848ddfdec07588" candidate_uuid: "4644b27d6e574a69ad298e4e0cab176c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a4338e83684a379c920b9fda7ebdbe" is_pre_election: true
I20260810 03:22:41.339919 3813 raft_consensus.cc:2781] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:22:41.340014 3732 raft_consensus.cc:2454] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4644b27d6e574a69ad298e4e0cab176c in term 0.
I20260810 03:22:41.340104 3813 raft_consensus.cc:482] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:22:41.340238 3813 raft_consensus.cc:3037] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:41.342792 3813 raft_consensus.cc:504] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.343590 3813 leader_election.cc:288] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [CANDIDATE]: Term 1 election: Requested vote from peers 0e25dc2714b745558b860e00d5a06a73 (127.19.246.193:46359), 24a4338e83684a379c920b9fda7ebdbe (127.19.246.195:39827)
I20260810 03:22:41.344298 3732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a15205d44584a7fbb848ddfdec07588" candidate_uuid: "4644b27d6e574a69ad298e4e0cab176c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a4338e83684a379c920b9fda7ebdbe"
I20260810 03:22:41.344298 3432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a15205d44584a7fbb848ddfdec07588" candidate_uuid: "4644b27d6e574a69ad298e4e0cab176c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e25dc2714b745558b860e00d5a06a73"
I20260810 03:22:41.344587 3732 raft_consensus.cc:3037] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:41.344664 3432 raft_consensus.cc:3037] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:41.348276 3432 raft_consensus.cc:2454] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4644b27d6e574a69ad298e4e0cab176c in term 1.
I20260810 03:22:41.348276 3732 raft_consensus.cc:2454] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4644b27d6e574a69ad298e4e0cab176c in term 1.
I20260810 03:22:41.349020 3507 leader_election.cc:302] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e25dc2714b745558b860e00d5a06a73, 24a4338e83684a379c920b9fda7ebdbe, 4644b27d6e574a69ad298e4e0cab176c; no voters:
I20260810 03:22:41.349490 3813 raft_consensus.cc:2781] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:22:41.350049 3813 raft_consensus.cc:686] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 1 LEADER]: Becoming Leader. State: Replica: 4644b27d6e574a69ad298e4e0cab176c, State: Running, Role: LEADER
I20260810 03:22:41.350656 3813 consensus_queue.cc:237] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:41.356387 3274 catalog_manager.cc:5495] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4644b27d6e574a69ad298e4e0cab176c (127.19.246.194). New cstate: current_term: 1 leader_uuid: "4644b27d6e574a69ad298e4e0cab176c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } health_report { overall_health: UNKNOWN } } }
W20260810 03:22:41.365175 3636 tablet.cc:2369] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:22:41.381418 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.375271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.19.246.194:41131)
0810 03:22:41.375524 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.375532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.375556 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:22:41.375826 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.375831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.375939 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.376142 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:22:41.376569 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.377531 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.379014 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.379516 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.379660 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.380211 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:22:41.380761 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:22:41.380933 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.381017 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.381110 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 03:22:41.391934 3432 raft_consensus.cc:1261] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 1 FOLLOWER]: Refusing update from remote peer 4644b27d6e574a69ad298e4e0cab176c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:22:41.391913 3732 raft_consensus.cc:1261] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 1 FOLLOWER]: Refusing update from remote peer 4644b27d6e574a69ad298e4e0cab176c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:22:41.393409 3813 consensus_queue.cc:1035] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:22:41.393838 3818 consensus_queue.cc:1035] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:22:41.412084 3824 mvcc.cc:205] Tried to move back new op lower bound from 7316816533048336384 to 7316816532898295808. Current Snapshot: MvccSnapshot[applied={T|T < 7316816533048336384}]
I20260810 03:22:41.426916 3275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:22:41.427295 3275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:22:41.431547 3275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Sending DeleteTablet for 3 replicas of tablet 8a15205d44584a7fbb848ddfdec07588
I20260810 03:22:41.433014 3411 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a15205d44584a7fbb848ddfdec07588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:41 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:54950
I20260810 03:22:41.433420 3709 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a15205d44584a7fbb848ddfdec07588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:41 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:47122
I20260810 03:22:41.433714 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 3340
I20260810 03:22:41.433888 3557 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a15205d44584a7fbb848ddfdec07588 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:41 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:58144
I20260810 03:22:41.445338 3834 tablet_replica.cc:330] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: stopping tablet replica
I20260810 03:22:41.446113 3834 raft_consensus.cc:2229] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:22:41.446852 3834 raft_consensus.cc:2258] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:41.447930 3836 tablet_replica.cc:330] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: stopping tablet replica
I20260810 03:22:41.450182 3834 ts_tablet_manager.cc:1905] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:22:41.450887 3833 tablet_replica.cc:330] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: stopping tablet replica
I20260810 03:22:41.451432 3833 raft_consensus.cc:2229] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:22:41.451843 3836 raft_consensus.cc:2229] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:22:41.452157 3833 raft_consensus.cc:2258] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:41.452595 3836 raft_consensus.cc:2258] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:41.455003 3836 ts_tablet_manager.cc:1905] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:22:41.458103 3833 ts_tablet_manager.cc:1905] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 03:22:41 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:22:41.463730 3836 ts_tablet_manager.cc:1918] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:41.464093 3836 log.cc:1196] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/8a15205d44584a7fbb848ddfdec07588
I20260810 03:22:41.464857 3836 ts_tablet_manager.cc:1939] T 8a15205d44584a7fbb848ddfdec07588 P 4644b27d6e574a69ad298e4e0cab176c: Deleting consensus metadata
I20260810 03:22:41.466163 3833 ts_tablet_manager.cc:1918] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:41.467175 3259 catalog_manager.cc:4841] TS 4644b27d6e574a69ad298e4e0cab176c (127.19.246.194:41131): tablet 8a15205d44584a7fbb848ddfdec07588 (table test-table [id=61856bb719a74f5dbc4cf8ff8b85938b]) successfully deleted
I20260810 03:22:41.469385 3833 log.cc:1196] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/8a15205d44584a7fbb848ddfdec07588
I20260810 03:22:41.469777 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.435295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.19.246.193:46359)
0810 03:22:41.435557 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.435566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.445460 (+ 9894us) server_negotiation.cc:407] Connection header received
0810 03:22:41.446107 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.446116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.446238 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.446374 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:41.449663 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.451017 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.453606 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.454298 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.454413 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.460446 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:41.460466 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:41.460493 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:41.460552 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:41.465197 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:41.466374 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:41.466384 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:41.466388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:41.466579 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:41.467189 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:41.467194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:41.467197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:41.467744 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:41.467992 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.468955 (+ 963us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.469224 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 03:22:41.470750 3476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:41.470799 3834 ts_tablet_manager.cc:1918] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:41.470989 3834 log.cc:1196] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/8a15205d44584a7fbb848ddfdec07588
I20260810 03:22:41.471704 3834 ts_tablet_manager.cc:1939] T 8a15205d44584a7fbb848ddfdec07588 P 24a4338e83684a379c920b9fda7ebdbe: Deleting consensus metadata
I20260810 03:22:41.474736 3255 catalog_manager.cc:4841] TS 24a4338e83684a379c920b9fda7ebdbe (127.19.246.195:39827): tablet 8a15205d44584a7fbb848ddfdec07588 (table test-table [id=61856bb719a74f5dbc4cf8ff8b85938b]) successfully deleted
I20260810 03:22:41.629719 3833 ts_tablet_manager.cc:1939] T 8a15205d44584a7fbb848ddfdec07588 P 0e25dc2714b745558b860e00d5a06a73: Deleting consensus metadata
I20260810 03:22:41.630448 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 3340
W20260810 03:22:41.657375 3256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.246.193:46359 (error 108)
I20260810 03:22:41.657660 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 3487
I20260810 03:22:41.660014 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.659157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:54972)
0810 03:22:41.659713 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.659834 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":301,"threads_started":1}
W20260810 03:22:41.660355 3256 catalog_manager.cc:4584] TS 0e25dc2714b745558b860e00d5a06a73 (127.19.246.193:46359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a15205d44584a7fbb848ddfdec07588: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Aug 10 03:22:41 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:22:41.678673 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.658520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.19.246.194:41131)
0810 03:22:41.658730 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.658737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.658943 (+ 206us) server_negotiation.cc:407] Connection header received
0810 03:22:41.659158 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.659164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.659286 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.659494 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:41.661445 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.662698 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.664859 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.665335 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.665450 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.671502 (+ 6052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:41.671522 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:41.671539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:41.671595 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:41.675304 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:41.676194 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:41.676219 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:41.676222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:41.676416 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:41.676869 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:41.676874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:41.676878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:41.677301 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:41.677692 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.678025 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.678292 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:22:41.679483 3630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:41.714394 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.713860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:54978)
0810 03:22:41.714142 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.714247 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260810 03:22:41.715713 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.715211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:54986)
0810 03:22:41.715430 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.715523 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 03:22:41.776523 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.775938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:54988)
0810 03:22:41.776176 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.776297 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 03:22:41.777806 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.777350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:55000)
0810 03:22:41.777513 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.777604 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 03:22:41.806571 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 3487
I20260810 03:22:41.829238 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 3641
Aug 10 03:22:41 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:22:41.847904 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.830137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.19.246.195:39827)
0810 03:22:41.830381 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:41.830385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:41.830404 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:22:41.830462 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:41.830466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:41.830529 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:41.830696 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:41.832141 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.832959 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.834365 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:41.834820 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:41.834934 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:41.840858 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:41.840876 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:41.840893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:41.840942 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:41.844347 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:41.845393 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:41.845401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:41.845404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:41.845564 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:41.846131 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:41.846136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:41.846138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:41.846601 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:41.846845 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:41.847181 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 03:22:41.847455 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260810 03:22:41.848735 3777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:41.858673 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.857943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:55006)
0810 03:22:41.858279 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.858421 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 03:22:41.860121 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:41.859547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:55020)
0810 03:22:41.859776 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:41.859893 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 03:22:41.947590 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 3641
I20260810 03:22:41.965749 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 3238
I20260810 03:22:41.967260 3309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:42.085939 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:42.085000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:55036)
0810 03:22:42.085465 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:42.085750 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 03:22:42.086223 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 3238
I20260810 03:22:42.114140 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41507
--webserver_interface=127.19.246.254
--webserver_port=43263
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:42.486438 3844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:42.486791 3844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:42.486955 3844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:42.495267 3844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:42.495559 3844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:42.495622 3844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:42.495685 3844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:42.495725 3844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:42.506331 3844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=43263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:42.507779 3844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:42.509125 3844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:42.540834 3854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:42.540745 3852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:42.540807 3851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:42.542527 3844 server_base.cc:1034] running on GCE node
I20260810 03:22:42.543744 3844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:42.545668 3844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:42.546949 3844 hybrid_clock.cc:648] HybridClock initialized: now 1786332162546887 us; error 59 us; skew 500 ppm
I20260810 03:22:42.547497 3844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:42 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332162, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:42.551965 3844 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:22:42.554129 3844 webserver.cc:460] Webserver started at http://127.19.246.254:43263/ using document root <none> and password file <none>
I20260810 03:22:42.555084 3844 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:42.555294 3844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:42.562601 3844 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:22:42.566548 3861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:42.568620 3844 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:22:42.568895 3844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "d71f82d4cc7243fe90d845cace809045"
format_stamp: "Formatted at 2026-08-10 03:22:38 on dist-test-slave-btw8"
server_key: "fc7ef2530cd0ebe52f605f3f5930987f"
server_key_iv: "cee226a676af4fa1df4115b775df8527"
server_key_version: "encryptionkey@0"
I20260810 03:22:42.570124 3844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:42.588392 3844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:42.593832 3844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:42.594199 3844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:42.617236 3844 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:41507
I20260810 03:22:42.617347 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:41507 every 8 connection(s)
I20260810 03:22:42.619045 3844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:22:42.619675 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 3844
I20260810 03:22:42.620733 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:46359
--local_ip_for_outbound_sockets=127.19.246.193
--tserver_master_addrs=127.19.246.254:41507
--webserver_port=46247
--webserver_interface=127.19.246.193
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:22:42.627285 3918 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:22:42.631035 3918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Bootstrap starting.
I20260810 03:22:42.653219 3918 log.cc:824] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:42.666705 3918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:22:42.667341 3918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Bootstrap complete.
I20260810 03:22:42.677721 3918 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:42.678684 3918 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71f82d4cc7243fe90d845cace809045, State: Initialized, Role: FOLLOWER
I20260810 03:22:42.679478 3918 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:42.679764 3918 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:22:42.679914 3918 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:22:42.680106 3918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:22:42.685089 3918 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:42.685801 3918 leader_election.cc:302] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71f82d4cc7243fe90d845cace809045; no voters:
I20260810 03:22:42.686539 3918 leader_election.cc:288] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:22:42.686722 3922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:22:42.688234 3922 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 2 LEADER]: Becoming Leader. State: Replica: d71f82d4cc7243fe90d845cace809045, State: Running, Role: LEADER
I20260810 03:22:42.688925 3922 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:42.689435 3918 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:42.691753 3924 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71f82d4cc7243fe90d845cace809045. Latest consensus state: current_term: 2 leader_uuid: "d71f82d4cc7243fe90d845cace809045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } } }
I20260810 03:22:42.692176 3924 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:42.692041 3923 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d71f82d4cc7243fe90d845cace809045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } } }
I20260810 03:22:42.692461 3923 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:42.702073 3929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:22:42.714694 3929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:22:42.716928 3929 catalog_manager.cc:1259] Loaded cluster ID: b8f5017771d8447f8bf1a466883f005a
I20260810 03:22:42.717151 3929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:22:42.727932 3929 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 03:22:42.746641 3929 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 03:22:42.756906 3929 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:22:42.758781 3929 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Loaded TSK: 0
I20260810 03:22:42.760596 3929 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:22:43.024559 3920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:43.024871 3920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:43.025035 3920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:43.032893 3920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:43.033031 3920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:43.033226 3920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:22:43.043892 3920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:46359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=46247
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:43.045498 3920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:43.046967 3920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:43.064173 3946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:43.064188 3948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:43.065575 3950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:43.065887 3920 server_base.cc:1034] running on GCE node
I20260810 03:22:43.066753 3920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:43.068037 3920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:43.069347 3920 hybrid_clock.cc:648] HybridClock initialized: now 1786332163069267 us; error 71 us; skew 500 ppm
I20260810 03:22:43.069792 3920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:43 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:43.074011 3920 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:22:43.076474 3920 webserver.cc:460] Webserver started at http://127.19.246.193:46247/ using document root <none> and password file <none>
I20260810 03:22:43.077590 3920 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:43.077829 3920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:43.084496 3920 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:22:43.088027 3957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:43.089902 3920 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:22:43.090117 3920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "0e25dc2714b745558b860e00d5a06a73"
format_stamp: "Formatted at 2026-08-10 03:22:39 on dist-test-slave-btw8"
server_key: "7275053c1e4a0d4d5e954a5150dd46c4"
server_key_iv: "2e5b2c5f23692a201de225fa92eddda2"
server_key_version: "encryptionkey@0"
I20260810 03:22:43.091464 3920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:43.124789 3920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:43.130458 3920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:43.130885 3920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:43.132257 3920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:43.134012 3920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:43.134166 3920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:43.134271 3920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:43.134357 3920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:43.173333 3920 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:46359
I20260810 03:22:43.173491 4082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:46359 every 8 connection(s)
I20260810 03:22:43.175203 3920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:22:43.178108 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 3920
I20260810 03:22:43.178838 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:41131
--local_ip_for_outbound_sockets=127.19.246.194
--tserver_master_addrs=127.19.246.254:41507
--webserver_port=38281
--webserver_interface=127.19.246.194
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:22:43 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:43.212695 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:43.179303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:33513)
0810 03:22:43.184187 (+ 4884us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:43.184267 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:43.190955 (+ 6688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:43.193666 (+ 2711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:43.193679 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:43.194374 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:43.195027 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:43.195050 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:43.197010 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:43.197021 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:43.198336 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:43.198352 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:43.199555 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:43.199564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:43.199863 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:43.200731 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:43.200757 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:43.203520 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:43.207636 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:43.207647 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:43.207665 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:43.208195 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:43.208754 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:43.208761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:43.208764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:43.208991 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:43.209891 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:43.209900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:43.210288 (+ 388us) client_negotiation.cc:769] Sending connection context
0810 03:22:43.210575 (+ 287us) client_negotiation.cc:240] Negotiation successful
0810 03:22:43.211227 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4474,"thread_start_us":4127,"threads_started":1}
I20260810 03:22:43.212774 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:43.180147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:33513 (local address 127.19.246.254:41507)
0810 03:22:43.181022 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:43.181050 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:43.184486 (+ 3436us) server_negotiation.cc:407] Connection header received
0810 03:22:43.192201 (+ 7715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:43.192223 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:43.192918 (+ 695us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:43.193455 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:43.195300 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:43.196806 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:43.198527 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:43.199213 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:43.199492 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:43.203950 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:43.203960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:43.203978 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:43.204023 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:43.207418 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:43.208386 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:43.208395 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:43.208400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:43.208611 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:43.209153 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:43.209158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:43.209163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:43.209729 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:43.209966 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:43.210809 (+ 843us) server_negotiation.cc:299] Negotiation successful
0810 03:22:43.211260 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":350,"threads_started":1}
I20260810 03:22:43.216069 4083 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41507
I20260810 03:22:43.216423 4083 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:43.217226 4083 heartbeater.cc:511] Master 127.19.246.254:41507 requested a full tablet report, sending...
I20260810 03:22:43.219755 3878 ts_manager.cc:194] Registered new tserver with Master: 0e25dc2714b745558b860e00d5a06a73 (127.19.246.193:46359)
I20260810 03:22:43.221324 3878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:33513
W20260810 03:22:43.553927 4088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:43.554306 4088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:43.554483 4088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:43.562218 4088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:43.562410 4088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:43.562625 4088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:22:43.573238 4088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:41131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=38281
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:43.574775 4088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:43.576308 4088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:43.592645 4094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:43.593048 4095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:43.592267 4097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:43.594969 4088 server_base.cc:1034] running on GCE node
I20260810 03:22:43.595955 4088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:43.597204 4088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:43.598577 4088 hybrid_clock.cc:648] HybridClock initialized: now 1786332163598522 us; error 41 us; skew 500 ppm
I20260810 03:22:43.599020 4088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:43 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:43.603003 4088 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:22:43.605090 4088 webserver.cc:460] Webserver started at http://127.19.246.194:38281/ using document root <none> and password file <none>
I20260810 03:22:43.605955 4088 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:43.606146 4088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:43.612861 4088 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:22:43.616423 4104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:43.618376 4088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:22:43.618613 4088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "4644b27d6e574a69ad298e4e0cab176c"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "001c441ba99ac7fe9142f4d7eea14b8b"
server_key_iv: "ab43654b4dc988b17784f4c87279e866"
server_key_version: "encryptionkey@0"
I20260810 03:22:43.620203 4088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:43.654983 4088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:43.660049 4088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:43.660334 4088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:43.661365 4088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:43.662830 4088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:43.662935 4088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:43.663012 4088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:43.663067 4088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:43.701772 4088 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:41131
I20260810 03:22:43.701906 4232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:41131 every 8 connection(s)
I20260810 03:22:43.703420 4088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:22:43.707342 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 4088
I20260810 03:22:43.708194 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:39827
--local_ip_for_outbound_sockets=127.19.246.195
--tserver_master_addrs=127.19.246.254:41507
--webserver_port=40553
--webserver_interface=127.19.246.195
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:22:43 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332163, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:43.730715 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:43.706421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:47181 (local address 127.19.246.254:41507)
0810 03:22:43.706655 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:43.706660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:43.708192 (+ 1532us) server_negotiation.cc:407] Connection header received
0810 03:22:43.710270 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:43.710279 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:43.710407 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:43.710547 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:43.712537 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:43.713691 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:43.715868 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:43.716855 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:43.717048 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:43.721824 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:43.721832 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:43.721835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:43.721877 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:43.725808 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:43.727507 (+ 1699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:43.727513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:43.727515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:43.727809 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:43.728468 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:43.728473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:43.728475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:43.728837 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:43.728984 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:43.730109 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0810 03:22:43.730377 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 03:22:43.732335 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:43.706960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:47181)
0810 03:22:43.707934 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:43.707987 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:43.709888 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:43.710834 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:43.710844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:43.711584 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:43.712276 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:43.712298 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:43.713911 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:43.713917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:43.715643 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:43.715669 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:43.717223 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:43.717229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:43.717484 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:43.718420 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:43.718446 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:43.721446 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:43.726113 (+ 4667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:43.726129 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:43.726157 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:43.727210 (+ 1053us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:43.727965 (+ 755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:43.727973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:43.727980 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:43.728307 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:43.729000 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:43.729011 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:43.729554 (+ 543us) client_negotiation.cc:769] Sending connection context
0810 03:22:43.729928 (+ 374us) client_negotiation.cc:240] Negotiation successful
0810 03:22:43.730589 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":440,"threads_started":1}
I20260810 03:22:43.734156 4233 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41507
I20260810 03:22:43.734591 4233 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:43.735512 4233 heartbeater.cc:511] Master 127.19.246.254:41507 requested a full tablet report, sending...
I20260810 03:22:43.737365 3878 ts_manager.cc:194] Registered new tserver with Master: 4644b27d6e574a69ad298e4e0cab176c (127.19.246.194:41131)
I20260810 03:22:43.738246 3878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:47181
W20260810 03:22:44.093030 4236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:44.093370 4236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:44.093565 4236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:44.102356 4236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:44.102563 4236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:44.102784 4236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:22:44.114665 4236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:39827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=40553
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:44.116183 4236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:44.117726 4236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:44.133554 4244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:44.133739 4243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:44.133739 4247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:44.135069 4236 server_base.cc:1034] running on GCE node
I20260810 03:22:44.136190 4236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:44.137400 4236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:44.138692 4236 hybrid_clock.cc:648] HybridClock initialized: now 1786332164138630 us; error 70 us; skew 500 ppm
I20260810 03:22:44.139053 4236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:44 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332164, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:44.142856 4236 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:22:44.144637 4236 webserver.cc:460] Webserver started at http://127.19.246.195:40553/ using document root <none> and password file <none>
I20260810 03:22:44.145398 4236 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:44.145586 4236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:44.151765 4236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:22:44.154897 4256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:44.156638 4236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:22:44.156860 4236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "24a4338e83684a379c920b9fda7ebdbe"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "537f2991f06d10fa13654ce4b637607f"
server_key_iv: "0bdcea3090319cbde42f917a4e907e4c"
server_key_version: "encryptionkey@0"
I20260810 03:22:44.157914 4236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:44.185982 4236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:44.191220 4236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:44.191577 4236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:44.192690 4236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:44.194186 4236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:44.194332 4236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:44.194486 4236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:44.194581 4236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:44.224815 4083 heartbeater.cc:503] Master 127.19.246.254:41507 was elected leader, sending a full tablet report...
I20260810 03:22:44.233832 4236 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:39827
I20260810 03:22:44.233896 4385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:39827 every 8 connection(s)
I20260810 03:22:44.235605 4236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:22:44.238236 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 4236
Aug 10 03:22:44 dist-test-slave-btw8 krb5kdc[3224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332164, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:22:44.257848 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.239243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.19.246.254:41507)
0810 03:22:44.239778 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.239786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.240213 (+ 427us) server_negotiation.cc:407] Connection header received
0810 03:22:44.240426 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.240432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.240530 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.240662 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:44.242835 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.243877 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.245989 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.246934 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.247110 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.251007 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:44.251012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:44.251015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:44.251057 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:44.254170 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.255235 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.255242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.255248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.255477 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.255986 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.255991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.255993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.256396 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:44.256742 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.257109 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.257346 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260810 03:22:44.262420 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.238700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:54847 (local address 127.19.246.254:41507)
0810 03:22:44.239227 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.239234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.240733 (+ 1499us) server_negotiation.cc:407] Connection header received
0810 03:22:44.243469 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.243475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.243561 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.243705 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:44.245989 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.247557 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.249651 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.250412 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.250566 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.254468 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:44.254477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:44.254483 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:44.254537 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:44.258321 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.259439 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.259445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.259448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.259648 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.260189 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.260193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.260196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.260552 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:44.260701 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.261786 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.262064 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":400,"thread_start_us":272,"threads_started":1}
I20260810 03:22:44.263536 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.239539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:54847)
0810 03:22:44.240553 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:44.240605 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:44.243157 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:44.244017 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:44.244031 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:44.244828 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:22:44.245685 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.245716 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.247745 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.247752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.249465 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.249481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.250655 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.250661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.250895 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.251712 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:22:44.251739 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:22:44.254241 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:22:44.258490 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:44.258502 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:44.258523 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:44.259148 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:44.259779 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:22:44.259784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:22:44.259787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:22:44.260063 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:22:44.260782 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:22:44.260794 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:22:44.261339 (+ 545us) client_negotiation.cc:769] Sending connection context
0810 03:22:44.261689 (+ 350us) client_negotiation.cc:240] Negotiation successful
0810 03:22:44.262234 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":423,"threads_started":1}
I20260810 03:22:44.265126 4386 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41507
I20260810 03:22:44.265506 4386 heartbeater.cc:464] Registering TS with master...
I20260810 03:22:44.266373 4386 heartbeater.cc:511] Master 127.19.246.254:41507 requested a full tablet report, sending...
I20260810 03:22:44.268399 3878 ts_manager.cc:194] Registered new tserver with Master: 24a4338e83684a379c920b9fda7ebdbe (127.19.246.195:39827)
I20260810 03:22:44.269295 3878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:54847
I20260810 03:22:44.274143 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:22:44.282251 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.275317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.19.246.254:41507)
0810 03:22:44.275551 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.275556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.276163 (+ 607us) server_negotiation.cc:407] Connection header received
0810 03:22:44.276375 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.276379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.276524 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.276664 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:22:44.277159 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.277881 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.279795 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.280283 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.280402 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.280999 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:22:44.281617 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:22:44.281738 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.281924 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.282027 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 03:22:44.283427 3878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:22:44.291483 3878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:22:44.333608 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.306798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.19.246.194:41131)
0810 03:22:44.323265 (+ 16467us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.323275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.323305 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:22:44.323457 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.323467 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.323888 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.324444 (+ 556us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:44.325359 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.327786 (+ 2427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.331119 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.332287 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.332510 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.332671 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.333039 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.333229 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16285,"thread_start_us":16168,"threads_started":1}
I20260810 03:22:44.333699 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.306143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:41131 (local address 127.0.0.1:58172)
0810 03:22:44.321709 (+ 15566us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:44.321738 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:44.321961 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:44.324227 (+ 2266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:44.324236 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:44.324271 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:44.325060 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.325076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.327963 (+ 2887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.327971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.330849 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.330868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.332532 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.332539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.332780 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.332824 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:22:44.332975 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 03:22:44.333226 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15353,"mutex_wait_us":8,"spinlock_wait_cycles":3840,"thread_start_us":17901,"threads_started":1}
I20260810 03:22:44.333774 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.307650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.19.246.195:39827)
0810 03:22:44.315826 (+ 8176us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.315841 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.324528 (+ 8687us) server_negotiation.cc:407] Connection header received
0810 03:22:44.324877 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.324883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.325050 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.325263 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:44.325976 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.327667 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.330460 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.332088 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.332286 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.332487 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.333200 (+ 713us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.333387 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7883,"thread_start_us":7569,"threads_started":1}
I20260810 03:22:44.334753 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.306242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.19.246.193:46359)
0810 03:22:44.323092 (+ 16850us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.323103 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.323139 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:22:44.323282 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.323296 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.323647 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.323943 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:44.325371 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.328025 (+ 2654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.331486 (+ 3461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.332965 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.333173 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.333327 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.334233 (+ 906us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.334415 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16624,"thread_start_us":16415,"threads_started":1}
I20260810 03:22:44.335094 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.306064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.0.0.1:55038)
0810 03:22:44.320744 (+ 14680us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:44.320836 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:44.321142 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:44.324268 (+ 3126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:44.324275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:44.324304 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:44.325060 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.325076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.328205 (+ 3129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.328214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.331276 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.331290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.333507 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.333520 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.333722 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.333764 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:22:44.333863 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 03:22:44.334046 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14228,"mutex_wait_us":76,"spinlock_wait_cycles":11520,"thread_start_us":15416,"threads_started":1}
I20260810 03:22:44.337390 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.306802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:39827 (local address 127.0.0.1:47146)
0810 03:22:44.324387 (+ 17585us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:44.324409 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:44.324666 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:44.325255 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:44.325261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:44.325288 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:44.325702 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.325713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.327887 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.327894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.330242 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.330261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.332731 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.332738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.332919 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.332952 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 03:22:44.333055 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 03:22:44.334276 (+ 1221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17423,"mutex_wait_us":1179,"spinlock_wait_cycles":896,"thread_start_us":15281,"threads_started":1}
I20260810 03:22:44.338229 4159 tablet_service.cc:1467] Processing CreateTablet for tablet 10620ca3f3a5460fa31748a968bc43e0 (DEFAULT_TABLE table=test-table [id=ca9363c5e7744ea788268c6df99f980a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:44.339031 4011 tablet_service.cc:1467] Processing CreateTablet for tablet 10620ca3f3a5460fa31748a968bc43e0 (DEFAULT_TABLE table=test-table [id=ca9363c5e7744ea788268c6df99f980a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:44.339792 4159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10620ca3f3a5460fa31748a968bc43e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:44.340605 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10620ca3f3a5460fa31748a968bc43e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:44.341234 4312 tablet_service.cc:1467] Processing CreateTablet for tablet 10620ca3f3a5460fa31748a968bc43e0 (DEFAULT_TABLE table=test-table [id=ca9363c5e7744ea788268c6df99f980a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:22:44.342998 4312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10620ca3f3a5460fa31748a968bc43e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:22:44.356500 4408 tablet_bootstrap.cc:492] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Bootstrap starting.
I20260810 03:22:44.360253 4409 tablet_bootstrap.cc:492] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Bootstrap starting.
I20260810 03:22:44.360566 4408 tablet_bootstrap.cc:654] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:44.362411 4408 log.cc:824] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:44.365958 4409 tablet_bootstrap.cc:654] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:44.367252 4408 tablet_bootstrap.cc:492] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: No bootstrap required, opened a new log
I20260810 03:22:44.367662 4408 ts_tablet_manager.cc:1397] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260810 03:22:44.367633 4409 log.cc:824] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:44.367944 4410 tablet_bootstrap.cc:492] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Bootstrap starting.
I20260810 03:22:44.371783 4409 tablet_bootstrap.cc:492] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: No bootstrap required, opened a new log
I20260810 03:22:44.371830 4410 tablet_bootstrap.cc:654] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Neither blocks nor log segments found. Creating new log.
I20260810 03:22:44.372131 4409 ts_tablet_manager.cc:1397] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260810 03:22:44.373090 4410 log.cc:824] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:44.377007 4410 tablet_bootstrap.cc:492] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: No bootstrap required, opened a new log
I20260810 03:22:44.377367 4410 ts_tablet_manager.cc:1397] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 03:22:44.380729 4408 raft_consensus.cc:348] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.381335 4408 raft_consensus.cc:374] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:44.381484 4408 raft_consensus.cc:729] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e25dc2714b745558b860e00d5a06a73, State: Initialized, Role: FOLLOWER
I20260810 03:22:44.382275 4408 consensus_queue.cc:260] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.384529 4408 ts_tablet_manager.cc:1428] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Time spent starting tablet: real 0.017s user 0.005s sys 0.010s
I20260810 03:22:44.384888 4409 raft_consensus.cc:348] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.385394 4409 raft_consensus.cc:374] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:44.385540 4409 raft_consensus.cc:729] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4644b27d6e574a69ad298e4e0cab176c, State: Initialized, Role: FOLLOWER
I20260810 03:22:44.386350 4409 consensus_queue.cc:260] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.387832 4233 heartbeater.cc:503] Master 127.19.246.254:41507 was elected leader, sending a full tablet report...
I20260810 03:22:44.387759 4410 raft_consensus.cc:348] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.388370 4410 raft_consensus.cc:374] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:22:44.388481 4410 raft_consensus.cc:729] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a4338e83684a379c920b9fda7ebdbe, State: Initialized, Role: FOLLOWER
I20260810 03:22:44.388688 4409 ts_tablet_manager.cc:1428] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260810 03:22:44.389340 4410 consensus_queue.cc:260] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.391340 4386 heartbeater.cc:503] Master 127.19.246.254:41507 was elected leader, sending a full tablet report...
I20260810 03:22:44.392241 4410 ts_tablet_manager.cc:1428] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260810 03:22:44.428093 4084 tablet.cc:2369] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:22:44.455730 4234 tablet.cc:2369] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:22:44.487670 4387 tablet.cc:2369] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:22:44.524438 4417 raft_consensus.cc:482] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:22:44.524811 4417 raft_consensus.cc:504] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.526800 4417 leader_election.cc:288] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e25dc2714b745558b860e00d5a06a73 (127.19.246.193:46359), 24a4338e83684a379c920b9fda7ebdbe (127.19.246.195:39827)
I20260810 03:22:44.534734 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.527230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:37961 (local address 127.19.246.193:46359)
0810 03:22:44.527506 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.527512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.528564 (+ 1052us) server_negotiation.cc:407] Connection header received
0810 03:22:44.528781 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.528788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.528880 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.528998 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:44.529697 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.530550 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.532660 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.533529 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.533668 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.533727 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.534296 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.534445 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:22:44.534755 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.527337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:49355 (local address 127.19.246.195:39827)
0810 03:22:44.527608 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.527614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.528762 (+ 1148us) server_negotiation.cc:407] Connection header received
0810 03:22:44.529018 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.529026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.529160 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.529294 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:22:44.530021 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.530870 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.532842 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.533695 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.533860 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.533937 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.534384 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.534524 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:22:44.534792 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.527405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46359 (local address 127.19.246.194:37961)
0810 03:22:44.528396 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:44.528422 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:44.528618 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:44.529061 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:44.529066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:44.529092 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:44.529525 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.529539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.530732 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.530737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.532470 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.532484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.533777 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.533787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.534005 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.534031 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:22:44.534129 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 03:22:44.534236 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":26,"thread_start_us":524,"threads_started":1}
I20260810 03:22:44.534972 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.527466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:39827 (local address 127.19.246.194:49355)
0810 03:22:44.528604 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:44.528629 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:22:44.528806 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:22:44.529339 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:22:44.529344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:22:44.529377 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:22:44.529849 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.529863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.531023 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.531026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.532669 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:22:44.532684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.533921 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.533926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:22:44.534074 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.534222 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 03:22:44.534331 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 03:22:44.534462 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":6,"thread_start_us":776,"threads_started":1}
I20260810 03:22:44.535949 4033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10620ca3f3a5460fa31748a968bc43e0" candidate_uuid: "4644b27d6e574a69ad298e4e0cab176c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e25dc2714b745558b860e00d5a06a73" is_pre_election: true
I20260810 03:22:44.536020 4339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10620ca3f3a5460fa31748a968bc43e0" candidate_uuid: "4644b27d6e574a69ad298e4e0cab176c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a4338e83684a379c920b9fda7ebdbe" is_pre_election: true
I20260810 03:22:44.536540 4033 raft_consensus.cc:2454] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4644b27d6e574a69ad298e4e0cab176c in term 0.
I20260810 03:22:44.536537 4339 raft_consensus.cc:2454] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4644b27d6e574a69ad298e4e0cab176c in term 0.
I20260810 03:22:44.537292 4106 leader_election.cc:302] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e25dc2714b745558b860e00d5a06a73, 4644b27d6e574a69ad298e4e0cab176c; no voters:
I20260810 03:22:44.537737 4417 raft_consensus.cc:2781] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:22:44.537916 4417 raft_consensus.cc:482] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:22:44.538051 4417 raft_consensus.cc:3037] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:44.540441 4417 raft_consensus.cc:504] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.541172 4417 leader_election.cc:288] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [CANDIDATE]: Term 1 election: Requested vote from peers 0e25dc2714b745558b860e00d5a06a73 (127.19.246.193:46359), 24a4338e83684a379c920b9fda7ebdbe (127.19.246.195:39827)
I20260810 03:22:44.541757 4033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10620ca3f3a5460fa31748a968bc43e0" candidate_uuid: "4644b27d6e574a69ad298e4e0cab176c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e25dc2714b745558b860e00d5a06a73"
I20260810 03:22:44.541796 4339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10620ca3f3a5460fa31748a968bc43e0" candidate_uuid: "4644b27d6e574a69ad298e4e0cab176c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a4338e83684a379c920b9fda7ebdbe"
I20260810 03:22:44.542001 4033 raft_consensus.cc:3037] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:44.542007 4339 raft_consensus.cc:3037] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:22:44.544399 4339 raft_consensus.cc:2454] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4644b27d6e574a69ad298e4e0cab176c in term 1.
I20260810 03:22:44.544405 4033 raft_consensus.cc:2454] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4644b27d6e574a69ad298e4e0cab176c in term 1.
I20260810 03:22:44.544919 4106 leader_election.cc:302] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e25dc2714b745558b860e00d5a06a73, 4644b27d6e574a69ad298e4e0cab176c; no voters:
I20260810 03:22:44.545259 4417 raft_consensus.cc:2781] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:22:44.545877 4417 raft_consensus.cc:686] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 1 LEADER]: Becoming Leader. State: Replica: 4644b27d6e574a69ad298e4e0cab176c, State: Running, Role: LEADER
I20260810 03:22:44.546403 4417 consensus_queue.cc:237] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } }
I20260810 03:22:44.551918 3878 catalog_manager.cc:5495] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4644b27d6e574a69ad298e4e0cab176c (127.19.246.194). New cstate: current_term: 1 leader_uuid: "4644b27d6e574a69ad298e4e0cab176c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4644b27d6e574a69ad298e4e0cab176c" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 } health_report { overall_health: UNKNOWN } } }
I20260810 03:22:44.581380 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.575569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.19.246.194:41131)
0810 03:22:44.575817 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.575822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.575842 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:22:44.575940 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.575944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.576027 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.576166 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:22:44.576550 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.577336 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.579017 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.579475 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.579580 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.580152 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:22:44.580802 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:22:44.580917 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.581064 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.581143 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:22:44.591084 4339 raft_consensus.cc:1261] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 1 FOLLOWER]: Refusing update from remote peer 4644b27d6e574a69ad298e4e0cab176c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:22:44.591096 4033 raft_consensus.cc:1261] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 1 FOLLOWER]: Refusing update from remote peer 4644b27d6e574a69ad298e4e0cab176c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:22:44.592681 4422 consensus_queue.cc:1035] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a4338e83684a379c920b9fda7ebdbe" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:22:44.593194 4417 consensus_queue.cc:1035] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e25dc2714b745558b860e00d5a06a73" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:22:44.607936 4426 mvcc.cc:205] Tried to move back new op lower bound from 7316816546151890944 to 7316816545986187264. Current Snapshot: MvccSnapshot[applied={T|T < 7316816546151890944}]
I20260810 03:22:44.626742 3877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:22:44.627206 3877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:22:44.632524 3877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Sending DeleteTablet for 3 replicas of tablet 10620ca3f3a5460fa31748a968bc43e0
I20260810 03:22:44.634253 4159 tablet_service.cc:1514] Processing DeleteTablet for tablet 10620ca3f3a5460fa31748a968bc43e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:44 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:58172
I20260810 03:22:44.635109 4011 tablet_service.cc:1514] Processing DeleteTablet for tablet 10620ca3f3a5460fa31748a968bc43e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:44 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:55038
I20260810 03:22:44.635301 4434 tablet_replica.cc:330] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: stopping tablet replica
I20260810 03:22:44.635874 4434 raft_consensus.cc:2229] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:22:44.636075 4435 tablet_replica.cc:330] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: stopping tablet replica
I20260810 03:22:44.636498 4435 raft_consensus.cc:2229] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:22:44.636737 4434 raft_consensus.cc:2258] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:44.636879 4312 tablet_service.cc:1514] Processing DeleteTablet for tablet 10620ca3f3a5460fa31748a968bc43e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:22:44 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:47146
I20260810 03:22:44.637130 4435 raft_consensus.cc:2258] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:44.638830 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 3920
I20260810 03:22:44.638947 4436 tablet_replica.cc:330] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: stopping tablet replica
I20260810 03:22:44.639367 4436 raft_consensus.cc:2229] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:22:44.640012 4436 raft_consensus.cc:2258] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:22:44.640905 4435 ts_tablet_manager.cc:1905] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:22:44.641209 4434 ts_tablet_manager.cc:1905] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:22:44.647172 4435 ts_tablet_manager.cc:1918] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:44.647536 4435 log.cc:1196] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/10620ca3f3a5460fa31748a968bc43e0
I20260810 03:22:44.648276 4435 ts_tablet_manager.cc:1939] T 10620ca3f3a5460fa31748a968bc43e0 P 0e25dc2714b745558b860e00d5a06a73: Deleting consensus metadata
I20260810 03:22:44.648751 4434 ts_tablet_manager.cc:1918] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:44.649057 4434 log.cc:1196] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/10620ca3f3a5460fa31748a968bc43e0
I20260810 03:22:44.649755 4434 ts_tablet_manager.cc:1939] T 10620ca3f3a5460fa31748a968bc43e0 P 4644b27d6e574a69ad298e4e0cab176c: Deleting consensus metadata
I20260810 03:22:44.650732 3863 catalog_manager.cc:4841] TS 0e25dc2714b745558b860e00d5a06a73 (127.19.246.193:46359): tablet 10620ca3f3a5460fa31748a968bc43e0 (table test-table [id=ca9363c5e7744ea788268c6df99f980a]) successfully deleted
I20260810 03:22:44.651718 3865 catalog_manager.cc:4841] TS 4644b27d6e574a69ad298e4e0cab176c (127.19.246.194:41131): tablet 10620ca3f3a5460fa31748a968bc43e0 (table test-table [id=ca9363c5e7744ea788268c6df99f980a]) successfully deleted
I20260810 03:22:44.652801 4436 ts_tablet_manager.cc:1905] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:22:44.658758 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.640133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.19.246.193:46359)
0810 03:22:44.640311 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.640317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.640338 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:22:44.640405 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.640410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.640503 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.640620 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:44.642718 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.643935 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.645834 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.646507 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.646639 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.650501 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:44.650516 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:44.650535 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:44.650577 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:44.654694 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.655866 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.655877 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.655881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.656075 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.656557 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.656562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.656565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.657127 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:44.657465 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.657867 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.658145 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:22:44.659858 4077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:44.659988 4436 ts_tablet_manager.cc:1918] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:22:44.660205 4436 log.cc:1196] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/10620ca3f3a5460fa31748a968bc43e0
I20260810 03:22:44.660691 4436 ts_tablet_manager.cc:1939] T 10620ca3f3a5460fa31748a968bc43e0 P 24a4338e83684a379c920b9fda7ebdbe: Deleting consensus metadata
I20260810 03:22:44.662479 3862 catalog_manager.cc:4841] TS 24a4338e83684a379c920b9fda7ebdbe (127.19.246.195:39827): tablet 10620ca3f3a5460fa31748a968bc43e0 (table test-table [id=ca9363c5e7744ea788268c6df99f980a]) successfully deleted
I20260810 03:22:44.776389 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 3920
I20260810 03:22:44.805662 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 4088
I20260810 03:22:44.827301 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.806510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.19.246.194:41131)
0810 03:22:44.806716 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.806722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.806744 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:22:44.806916 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.806922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.807034 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.807163 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:44.808651 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.809918 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.811525 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.812005 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.812173 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.819939 (+ 7766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:44.819955 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:44.819970 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:44.820035 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:44.823853 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.824813 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.824821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.824823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.824994 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.825431 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.825443 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.825445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.825843 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:44.826261 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.826645 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.826917 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:22:44.828280 4227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:44.954276 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 4088
I20260810 03:22:44.978826 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 4236
I20260810 03:22:44.996137 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:44.979666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.19.246.195:39827)
0810 03:22:44.979908 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 03:22:44.979915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:22:44.980066 (+ 151us) server_negotiation.cc:407] Connection header received
0810 03:22:44.980300 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:22:44.980306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:22:44.980389 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:22:44.980518 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:22:44.982067 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.982839 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.984346 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:22:44.984774 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:22:44.984938 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:22:44.988137 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:22:44.988155 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:22:44.988181 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:22:44.988223 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:22:44.992246 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.993310 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.993321 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.993323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.993529 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:22:44.994032 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:22:44.994040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:22:44.994051 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:22:44.994513 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:22:44.994917 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0810 03:22:44.995291 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 03:22:44.995610 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:22:44.997009 4381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:45.091409 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 4236
I20260810 03:22:45.111907 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 3844
I20260810 03:22:45.113255 3911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:22:45.236123 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 3844
I20260810 03:22:45.263340 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41507
--webserver_interface=127.19.246.254
--webserver_port=43263
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:45.647606 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:45.647953 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:45.648141 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:45.657017 4441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:22:45.657184 4441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:45.657296 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:45.657401 4441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:22:45.657475 4441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:22:45.669754 4441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=43263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:45.671525 4441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:45.673120 4441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:45.687482 4447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:45.687497 4448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:45.687781 4450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:45.689325 4441 server_base.cc:1034] running on GCE node
I20260810 03:22:45.690447 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:45.692523 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:45.693831 4441 hybrid_clock.cc:648] HybridClock initialized: now 1786332165693762 us; error 69 us; skew 500 ppm
I20260810 03:22:45.694276 4441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:45 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332165, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:45.698454 4441 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:22:45.700505 4441 webserver.cc:460] Webserver started at http://127.19.246.254:43263/ using document root <none> and password file <none>
I20260810 03:22:45.701367 4441 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:45.701539 4441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:45.708629 4441 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:22:45.714700 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:45.716578 4441 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:22:45.716867 4441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "d71f82d4cc7243fe90d845cace809045"
format_stamp: "Formatted at 2026-08-10 03:22:38 on dist-test-slave-btw8"
server_key: "fc7ef2530cd0ebe52f605f3f5930987f"
server_key_iv: "cee226a676af4fa1df4115b775df8527"
server_key_version: "encryptionkey@0"
I20260810 03:22:45.718055 4441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:45.748064 4441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:45.753520 4441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:45.753870 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:45.776569 4441 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:41507
I20260810 03:22:45.776628 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:41507 every 8 connection(s)
I20260810 03:22:45.778429 4441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:22:45.786646 4516 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:22:45.788062 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 4441
I20260810 03:22:45.789196 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:46359
--local_ip_for_outbound_sockets=127.19.246.193
--tserver_master_addrs=127.19.246.254:41507
--webserver_port=46247
--webserver_interface=127.19.246.193
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:22:45.791013 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Bootstrap starting.
I20260810 03:22:45.818822 4516 log.cc:824] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Log is configured to *not* fsync() on all Append() calls
I20260810 03:22:45.844031 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:22:45.844799 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045: Bootstrap complete.
I20260810 03:22:45.859068 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:45.860584 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71f82d4cc7243fe90d845cace809045, State: Initialized, Role: FOLLOWER
I20260810 03:22:45.861416 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:45.861703 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:22:45.861845 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:22:45.862031 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:22:45.868502 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:45.869248 4516 leader_election.cc:302] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71f82d4cc7243fe90d845cace809045; no voters:
I20260810 03:22:45.870162 4516 leader_election.cc:288] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 03:22:45.870347 4520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:22:45.872112 4520 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [term 3 LEADER]: Becoming Leader. State: Replica: d71f82d4cc7243fe90d845cace809045, State: Running, Role: LEADER
I20260810 03:22:45.872867 4520 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } }
I20260810 03:22:45.873272 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:22:45.875900 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d71f82d4cc7243fe90d845cace809045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } } }
I20260810 03:22:45.875903 4522 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71f82d4cc7243fe90d845cace809045. Latest consensus state: current_term: 3 leader_uuid: "d71f82d4cc7243fe90d845cace809045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f82d4cc7243fe90d845cace809045" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41507 } } }
I20260810 03:22:45.876446 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:45.876441 4522 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71f82d4cc7243fe90d845cace809045 [sys.catalog]: This master's current role is: LEADER
I20260810 03:22:45.877929 4525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:22:45.896047 4525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:22:45.898629 4525 catalog_manager.cc:1259] Loaded cluster ID: b8f5017771d8447f8bf1a466883f005a
I20260810 03:22:45.898864 4525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 03:22:45.901897 4525 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1740ee59ad google::LogMessage::Fail() at ??:0
@ 0x7f1740ee888a google::LogMessage::SendToLog() at ??:0
@ 0x7f1740ee54de google::LogMessage::Flush() at ??:0
@ 0x7f1740ee6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1756918512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1756915976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f175699d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f175699d439 std::_Function_handler<>::_M_invoke() at ??:0
I20260810 03:22:45.937387 4539 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1743b31345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1743b3491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1743b34799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1743b0feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1746de96db start_thread at ??:0
@ 0x7f173fd7f61f clone at ??:0
W20260810 03:22:46.270247 4518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:46.270617 4518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:46.270812 4518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:46.280117 4518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:46.280257 4518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:46.280470 4518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:22:46.293027 4518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:46359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=46247
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:46.294595 4518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:46.296180 4518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:46.313066 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:46.313295 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:46.313242 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:46.314580 4518 server_base.cc:1034] running on GCE node
I20260810 03:22:46.315562 4518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:46.316758 4518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:46.317993 4518 hybrid_clock.cc:648] HybridClock initialized: now 1786332166317951 us; error 51 us; skew 500 ppm
I20260810 03:22:46.318373 4518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:46 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332166, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:46.322151 4518 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:22:46.324556 4518 webserver.cc:460] Webserver started at http://127.19.246.193:46247/ using document root <none> and password file <none>
I20260810 03:22:46.325590 4518 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:46.325829 4518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:46.333101 4518 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:22:46.337186 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:46.339728 4518 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260810 03:22:46.340004 4518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "0e25dc2714b745558b860e00d5a06a73"
format_stamp: "Formatted at 2026-08-10 03:22:39 on dist-test-slave-btw8"
server_key: "7275053c1e4a0d4d5e954a5150dd46c4"
server_key_iv: "2e5b2c5f23692a201de225fa92eddda2"
server_key_version: "encryptionkey@0"
I20260810 03:22:46.341524 4518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:46.366428 4518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:46.371727 4518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:46.371999 4518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:46.373200 4518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:46.374864 4518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:46.374967 4518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:46.375144 4518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:46.375236 4518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:46.418144 4518 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:46359
I20260810 03:22:46.418210 4697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:46359 every 8 connection(s)
I20260810 03:22:46.419839 4518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:22:46.422967 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 4518
I20260810 03:22:46.423789 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:41131
--local_ip_for_outbound_sockets=127.19.246.194
--tserver_master_addrs=127.19.246.254:41507
--webserver_port=38281
--webserver_interface=127.19.246.194
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:22:46.426057 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:46.423396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:36985)
0810 03:22:46.424412 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:46.424753 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":369,"threads_started":1}
W20260810 03:22:46.426968 4698 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:41507 (0 consecutive failures): Network error: Failed to ping master at 127.19.246.254:41507: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
I20260810 03:22:46.428238 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:46.427755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:59981)
0810 03:22:46.427963 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:46.428080 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 03:22:46.429410 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:46.429007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:34407)
0810 03:22:46.429179 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:46.429281 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 03:22:46.429783 4698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:22:46.799296 4702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:46.799718 4702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:46.799927 4702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:46.807962 4702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:46.808089 4702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:46.808279 4702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:22:46.818843 4702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:41131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=38281
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:46.820420 4702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:46.821866 4702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:46.837266 4709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:46.837244 4711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:46.837652 4708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:46.838176 4702 server_base.cc:1034] running on GCE node
I20260810 03:22:46.838965 4702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:46.840255 4702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:46.841537 4702 hybrid_clock.cc:648] HybridClock initialized: now 1786332166841467 us; error 67 us; skew 500 ppm
I20260810 03:22:46.841948 4702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:46 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332166, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:46.845709 4702 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:22:46.847637 4702 webserver.cc:460] Webserver started at http://127.19.246.194:38281/ using document root <none> and password file <none>
I20260810 03:22:46.848526 4702 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:46.848701 4702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:46.854836 4702 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 03:22:46.858131 4718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:46.859817 4702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 03:22:46.860028 4702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "4644b27d6e574a69ad298e4e0cab176c"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "001c441ba99ac7fe9142f4d7eea14b8b"
server_key_iv: "ab43654b4dc988b17784f4c87279e866"
server_key_version: "encryptionkey@0"
I20260810 03:22:46.861138 4702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:46.876232 4702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:46.880939 4702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:46.881270 4702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:46.882396 4702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:46.884066 4702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:46.884164 4702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:46.884305 4702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:46.884428 4702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:46.922644 4702 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:41131
I20260810 03:22:46.922781 4843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:41131 every 8 connection(s)
I20260810 03:22:46.924400 4702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:22:46.930107 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 4702
I20260810 03:22:46.930389 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:46.928019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:50005)
0810 03:22:46.928911 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:46.929181 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":393,"threads_started":1}
I20260810 03:22:46.930929 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:39827
--local_ip_for_outbound_sockets=127.19.246.195
--tserver_master_addrs=127.19.246.254:41507
--webserver_port=40553
--webserver_interface=127.19.246.195
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:22:46.931310 4844 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:41507 (0 consecutive failures): Network error: Failed to ping master at 127.19.246.254:41507: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
I20260810 03:22:46.932628 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:46.932049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:33369)
0810 03:22:46.932321 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:46.932434 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 03:22:46.934016 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:46.933663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:39867)
0810 03:22:46.933812 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:46.933895 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 03:22:46.934331 4844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:22:47.293697 4847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:22:47.294080 4847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:22:47.294266 4847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:22:47.301863 4847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:22:47.302057 4847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:22:47.302282 4847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:22:47.312808 4847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:39827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=40553
--tserver_master_addrs=127.19.246.254:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:22:47.314291 4847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:22:47.315827 4847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:22:47.330554 4854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:47.330682 4856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:22:47.330749 4853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:22:47.331528 4847 server_base.cc:1034] running on GCE node
I20260810 03:22:47.332372 4847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:22:47.333729 4847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:22:47.335034 4847 hybrid_clock.cc:648] HybridClock initialized: now 1786332167334964 us; error 71 us; skew 500 ppm
I20260810 03:22:47.335551 4847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:22:47 dist-test-slave-btw8 krb5kdc[3224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332167, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:22:47.339295 4847 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:22:47.341159 4847 webserver.cc:460] Webserver started at http://127.19.246.195:40553/ using document root <none> and password file <none>
I20260810 03:22:47.341966 4847 fs_manager.cc:359] Metadata directory not provided
I20260810 03:22:47.342145 4847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:22:47.348933 4847 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:22:47.352943 4863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:47.355049 4847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:22:47.355523 4847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "24a4338e83684a379c920b9fda7ebdbe"
format_stamp: "Formatted at 2026-08-10 03:22:40 on dist-test-slave-btw8"
server_key: "537f2991f06d10fa13654ce4b637607f"
server_key_iv: "0bdcea3090319cbde42f917a4e907e4c"
server_key_version: "encryptionkey@0"
I20260810 03:22:47.356709 4847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:22:47.372015 4847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:22:47.377104 4847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:22:47.377440 4847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:22:47.378522 4847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:22:47.380110 4847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:22:47.380244 4847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:47.380386 4847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:22:47.380478 4847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:22:47.418557 4847 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:39827
I20260810 03:22:47.418601 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:39827 every 8 connection(s)
I20260810 03:22:47.420668 4847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:22:47.425312 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 4847
I20260810 03:22:47.426303 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:47.424127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:36727)
0810 03:22:47.424892 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:47.425122 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":298,"threads_started":1}
W20260810 03:22:47.427217 4985 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:41507 (0 consecutive failures): Network error: Failed to ping master at 127.19.246.254:41507: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
I20260810 03:22:47.428143 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:47.427811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:34491)
0810 03:22:47.427958 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:47.428027 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 03:22:47.428988 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:47.428641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:57671)
0810 03:22:47.428799 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:47.428882 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 03:22:47.429268 4985 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:22:47.431262 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:47.430314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:40033)
0810 03:22:47.430754 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:47.430911 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":210,"threads_started":1}
I20260810 03:22:47.936062 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:47.935037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:43031)
0810 03:22:47.935601 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:47.935783 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":253,"threads_started":1}
I20260810 03:22:48.431048 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:48.430040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:37381)
0810 03:22:48.430616 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:48.430786 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":302,"threads_started":1}
I20260810 03:22:48.433236 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:48.432354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:38987)
0810 03:22:48.432857 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:48.432995 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":237,"threads_started":1}
I20260810 03:22:48.938277 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:48.937385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:55451)
0810 03:22:48.937958 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:48.938086 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":294,"threads_started":1}
I20260810 03:22:49.433169 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:49.432247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:43107)
0810 03:22:49.432873 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:49.432971 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":307,"threads_started":1}
I20260810 03:22:49.435696 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:49.434692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:51195)
0810 03:22:49.435264 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:49.435408 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":307,"threads_started":1}
I20260810 03:22:49.940351 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:49.939495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:45027)
0810 03:22:49.940047 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:49.940151 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":296,"threads_started":1}
I20260810 03:22:50.435451 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:50.434266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:49869)
0810 03:22:50.434917 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:50.435121 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":444,"threads_started":1}
I20260810 03:22:50.437616 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:50.436799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:53139)
0810 03:22:50.437358 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:50.437441 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":296,"threads_started":1}
I20260810 03:22:50.942233 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:50.941395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:54401)
0810 03:22:50.941917 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:50.942010 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":262,"threads_started":1}
I20260810 03:22:51.437989 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:51.436724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:55733)
0810 03:22:51.437310 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:51.437407 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":319,"threads_started":1}
I20260810 03:22:51.439918 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:51.438956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:59803)
0810 03:22:51.439617 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:51.439745 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":336,"threads_started":1}
I20260810 03:22:51.944334 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:51.943459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:37793)
0810 03:22:51.944045 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:51.944135 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":343,"threads_started":1}
I20260810 03:22:52.440446 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:52.439657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:58113)
0810 03:22:52.440171 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:52.440262 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
I20260810 03:22:52.441983 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:52.441156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:53207)
0810 03:22:52.441716 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:52.441822 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":321,"threads_started":1}
I20260810 03:22:52.946278 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:52.945331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:36415)
0810 03:22:52.945982 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:52.946092 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":427,"threads_started":1}
I20260810 03:22:53.442667 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:53.441691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:36171)
0810 03:22:53.442313 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:53.442442 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":350,"threads_started":1}
I20260810 03:22:53.444104 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:53.443005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:60407)
0810 03:22:53.443814 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:53.443899 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":426,"threads_started":1}
I20260810 03:22:53.948311 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:53.947436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:53485)
0810 03:22:53.947987 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:53.948084 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":244,"threads_started":1}
I20260810 03:22:54.444800 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:54.443916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:45805)
0810 03:22:54.444419 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:54.444514 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":225,"threads_started":1}
I20260810 03:22:54.446066 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:54.445168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:36989)
0810 03:22:54.445819 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:54.445915 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":304,"threads_started":1}
I20260810 03:22:54.950160 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:54.949403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:55089)
0810 03:22:54.949887 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:54.949977 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260810 03:22:55.446749 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:55.445848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:35005)
0810 03:22:55.446471 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:55.446560 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":396,"threads_started":1}
I20260810 03:22:55.447835 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:55.446985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:49415)
0810 03:22:55.447579 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:55.447665 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
I20260810 03:22:55.952277 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:55.951264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:38053)
0810 03:22:55.951962 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:55.952063 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":362,"threads_started":1}
I20260810 03:22:56.449097 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:56.447902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:50557)
0810 03:22:56.448615 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:56.448823 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":365,"threads_started":1}
I20260810 03:22:56.449858 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:56.448882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:42799)
0810 03:22:56.449594 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:56.449689 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":522,"threads_started":1}
I20260810 03:22:56.954375 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:56.953513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:57773)
0810 03:22:56.954065 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:56.954200 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":306,"threads_started":1}
I20260810 03:22:57.451539 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:57.450452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:40299)
0810 03:22:57.451105 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:57.451253 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":364,"threads_started":1}
I20260810 03:22:57.451961 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:57.450991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:36361)
0810 03:22:57.451638 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:57.451768 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":315,"threads_started":1}
I20260810 03:22:57.956418 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:57.955622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:51563)
0810 03:22:57.956096 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:57.956211 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":240,"threads_started":1}
I20260810 03:22:58.453624 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:58.452693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:38805)
0810 03:22:58.453239 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:58.453393 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":263,"threads_started":1}
I20260810 03:22:58.453982 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:58.453098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:53197)
0810 03:22:58.453632 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:58.453730 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":250,"threads_started":1}
I20260810 03:22:58.958698 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:58.957719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:48589)
0810 03:22:58.958315 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:58.958421 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":309,"threads_started":1}
I20260810 03:22:59.455727 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:59.454759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:46591)
0810 03:22:59.455326 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:59.455497 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":375,"threads_started":1}
I20260810 03:22:59.456178 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:59.454920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:43401)
0810 03:22:59.455821 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:59.455955 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":582,"threads_started":1}
I20260810 03:22:59.960868 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:22:59.959857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:34107)
0810 03:22:59.960378 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 03:22:59.960557 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":280,"threads_started":1}
I20260810 03:23:00.458097 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:00.457021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:34267)
0810 03:23:00.457668 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:00.457806 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":340,"threads_started":1}
I20260810 03:23:00.458258 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:00.457255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:59369)
0810 03:23:00.457919 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:00.458029 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":433,"threads_started":1}
I20260810 03:23:00.962895 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:00.962083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:45901)
0810 03:23:00.962608 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:00.962704 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":263,"threads_started":1}
I20260810 03:23:01.460501 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:01.459346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:57423)
0810 03:23:01.460048 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:01.460196 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":382,"threads_started":1}
I20260810 03:23:01.460793 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:01.459596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:44067)
0810 03:23:01.460399 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:01.460549 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":574,"threads_started":1}
I20260810 03:23:01.965224 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:01.964272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.194:35641)
0810 03:23:01.964807 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:01.964927 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":256,"threads_started":1}
I20260810 03:23:02.434053 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 4518
I20260810 03:23:02.461969 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:02.435482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.19.246.193:46359)
0810 03:23:02.436268 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:02.436292 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:02.436327 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:23:02.438027 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:02.438038 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:02.438511 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:02.438893 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:02.440578 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:02.441859 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:02.443228 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:02.444005 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:02.444224 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:02.448757 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:02.448773 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:02.448822 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:02.448906 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:02.455290 (+ 6384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:02.457108 (+ 1818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:02.457122 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:02.457137 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:02.457435 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:02.458033 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:02.458038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:02.458042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:02.458940 (+ 898us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:02.460518 (+ 1578us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:02.460884 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 03:23:02.461404 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":434,"threads_started":1}
I20260810 03:23:02.464255 4691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:02.472469 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:02.461777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.195:58479)
0810 03:23:02.471956 (+ 10179us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:02.472139 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9950,"thread_start_us":9827,"threads_started":1}
I20260810 03:23:02.580864 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:02.461743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41507 (local address 127.19.246.193:49917)
0810 03:23:02.580351 (+118608us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:02.580568 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118451,"thread_start_us":118244,"threads_started":1}
I20260810 03:23:02.581322 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 4518
I20260810 03:23:02.602329 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 4702
I20260810 03:23:02.630622 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:02.603621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.19.246.194:41131)
0810 03:23:02.604178 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:02.604192 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:02.604229 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:23:02.605817 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:02.605827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:02.606307 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:02.606701 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:02.609192 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:02.612139 (+ 2947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:02.615091 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:02.616804 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:02.618020 (+ 1216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:02.622723 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:02.622742 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:02.622775 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:02.622863 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:02.626370 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:02.627468 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:02.627477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:02.627485 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:02.627719 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:02.628261 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:02.628265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:02.628267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:02.628988 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:02.629257 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:02.629773 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 03:23:02.630241 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":220,"threads_started":1}
I20260810 03:23:02.632293 4838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:02.719915 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 4702
I20260810 03:23:02.741470 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 4847
I20260810 03:23:02.763295 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:02.742751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.19.246.195:39827)
0810 03:23:02.743490 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:02.743506 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:02.743552 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:23:02.745264 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:02.745275 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:02.745750 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:02.746156 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:02.747983 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:02.749327 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:02.750894 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:02.751785 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:02.752037 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:02.755856 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:02.755866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:02.755886 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:02.755938 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:02.759249 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:02.760241 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:02.760250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:02.760258 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:02.760521 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:02.760971 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:02.760976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:02.760981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:02.761610 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:02.761990 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:02.762477 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 03:23:02.762958 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":320,"threads_started":1}
I20260810 03:23:02.764926 4980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:02.854892 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 4847
2026-08-10T03:23:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25949 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:23:02 dist-test-slave-btw8 krb5kdc[5051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:23:02 dist-test-slave-btw8 krb5kdc[5051](info): set up 2 sockets
Aug 10 03:23:02 dist-test-slave-btw8 krb5kdc[5051](info): commencing operation
krb5kdc: starting...
W20260810 03:23:04.082226 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.002s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:23:04 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:23:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:23:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:23:04.432101 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36501
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:23:04.800825 5065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:04.801235 5065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:04.801410 5065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:04.810616 5065 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:23:04.810794 5065 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:23:04.810894 5065 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:23:04.810981 5065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:23:04.811069 5065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:04.811178 5065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:04.811311 5065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:23:04.811442 5065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:23:04.822122 5065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:04.823745 5065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:04.825350 5065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:04.838845 5070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:04.838752 5073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:04.838819 5071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:04.840098 5065 server_base.cc:1034] running on GCE node
I20260810 03:23:04.841192 5065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:04.842847 5065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:04.844141 5065 hybrid_clock.cc:648] HybridClock initialized: now 1786332184844066 us; error 64 us; skew 500 ppm
I20260810 03:23:04.844547 5065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:04 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332184, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:04.855201 5065 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:23:04.857262 5065 webserver.cc:460] Webserver started at http://127.19.246.254:40991/ using document root <none> and password file <none>
I20260810 03:23:04.858112 5065 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:04.858222 5065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:04.858528 5065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:23:04.862087 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "52e84229034a4933bc0c2854b206f8c8"
format_stamp: "Formatted at 2026-08-10 03:23:04 on dist-test-slave-btw8"
server_key: "7d70dd2e6f7f3d08880bc5450234b576"
server_key_iv: "3b1dd65161dfedcd49955e40539c46a6"
server_key_version: "encryptionkey@0"
I20260810 03:23:04.862941 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "52e84229034a4933bc0c2854b206f8c8"
format_stamp: "Formatted at 2026-08-10 03:23:04 on dist-test-slave-btw8"
server_key: "7d70dd2e6f7f3d08880bc5450234b576"
server_key_iv: "3b1dd65161dfedcd49955e40539c46a6"
server_key_version: "encryptionkey@0"
I20260810 03:23:04.869177 5065 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 03:23:04.873569 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:04.875288 5065 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 03:23:04.875597 5065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "52e84229034a4933bc0c2854b206f8c8"
format_stamp: "Formatted at 2026-08-10 03:23:04 on dist-test-slave-btw8"
server_key: "7d70dd2e6f7f3d08880bc5450234b576"
server_key_iv: "3b1dd65161dfedcd49955e40539c46a6"
server_key_version: "encryptionkey@0"
I20260810 03:23:04.875955 5065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:04.899060 5065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:04.904222 5065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:04.904577 5065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:04.927826 5065 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:36501
I20260810 03:23:04.927841 5140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:36501 every 8 connection(s)
I20260810 03:23:04.929519 5065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:23:04.934149 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:23:04.937633 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 5065
I20260810 03:23:04.937913 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:23:04.938921 20443 external_mini_cluster.cc:1428] Setting key 575af70445551722a221ef6f281e9f5c
I20260810 03:23:04.949798 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Bootstrap starting.
I20260810 03:23:04.954708 5141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Neither blocks nor log segments found. Creating new log.
I20260810 03:23:04.956507 5141 log.cc:824] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:23:04 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:23:04.960659 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: No bootstrap required, opened a new log
I20260810 03:23:04.968959 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:04.941624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.19.246.254:36501)
0810 03:23:04.942529 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:04.942543 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:04.942594 (+ 51us) server_negotiation.cc:407] Connection header received
0810 03:23:04.944702 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:04.944732 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:04.945380 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:04.945790 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:04.948099 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:04.949625 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:04.951746 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:04.952700 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:04.952884 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:04.959523 (+ 6639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:04.959537 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:04.959555 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:04.959602 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:04.963729 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:04.964719 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:04.964728 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:04.964736 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:04.964948 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:04.965417 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:04.965421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:04.965427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:04.966102 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:04.966508 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:04.967040 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 03:23:04.967601 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":433,"threads_started":1}
I20260810 03:23:04.970232 5141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:04.970743 5141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:23:04.970898 5141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e84229034a4933bc0c2854b206f8c8, State: Initialized, Role: FOLLOWER
I20260810 03:23:04.971688 5141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:04.972028 5141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:23:04.972246 5141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:23:04.972493 5141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:23:04.975157 5141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:04.975925 5141 leader_election.cc:302] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e84229034a4933bc0c2854b206f8c8; no voters:
I20260810 03:23:04.976718 5141 leader_election.cc:288] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:23:04.976863 5146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:23:04.978389 5146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 1 LEADER]: Becoming Leader. State: Replica: 52e84229034a4933bc0c2854b206f8c8, State: Running, Role: LEADER
I20260810 03:23:04.979015 5146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:04.979249 5141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:23:04.982524 5147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e84229034a4933bc0c2854b206f8c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } } }
I20260810 03:23:04.983101 5147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:23:04.983615 5148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e84229034a4933bc0c2854b206f8c8. Latest consensus state: current_term: 1 leader_uuid: "52e84229034a4933bc0c2854b206f8c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } } }
I20260810 03:23:04.984040 5148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: This master's current role is: LEADER
W20260810 03:23:04.996423 5163 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:23:04.996574 5163 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:23:04.996742 5154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:23:04.999249 5154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:23:05.009312 5154 catalog_manager.cc:1347] Generated new cluster ID: 61871238f04346a298cecd6283bb0c12
I20260810 03:23:05.009454 5154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:23:05.024825 5154 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:23:05.027222 5154 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:23:05.035590 5154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Generated new TSK 0
I20260810 03:23:05.036625 5154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:23:05.132768 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36501
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:23:05.521123 5169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:05.521450 5169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:05.521627 5169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:05.529634 5169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:05.529791 5169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:05.529996 5169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:23:05.541029 5169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:05.542491 5169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:05.544010 5169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:05.558560 5176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:05.558523 5174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:05.558523 5178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:05.559911 5169 server_base.cc:1034] running on GCE node
I20260810 03:23:05.560632 5169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:05.561766 5169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:05.563102 5169 hybrid_clock.cc:648] HybridClock initialized: now 1786332185563030 us; error 35 us; skew 500 ppm
I20260810 03:23:05.563562 5169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:05 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332185, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:05.567274 5169 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:23:05.569213 5169 webserver.cc:460] Webserver started at http://127.19.246.193:40655/ using document root <none> and password file <none>
I20260810 03:23:05.569996 5169 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:05.570116 5169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:05.570415 5169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:23:05.573881 5169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "18edfbfb048e4b94892801df59ea2da7"
format_stamp: "Formatted at 2026-08-10 03:23:05 on dist-test-slave-btw8"
server_key: "5700d0f985c774fe9612a0962a5e45bd"
server_key_iv: "d4fc97956d41d2870ccfeb8ddaceca0f"
server_key_version: "encryptionkey@0"
I20260810 03:23:05.574720 5169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "18edfbfb048e4b94892801df59ea2da7"
format_stamp: "Formatted at 2026-08-10 03:23:05 on dist-test-slave-btw8"
server_key: "5700d0f985c774fe9612a0962a5e45bd"
server_key_iv: "d4fc97956d41d2870ccfeb8ddaceca0f"
server_key_version: "encryptionkey@0"
I20260810 03:23:05.580969 5169 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260810 03:23:05.584860 5188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:05.586385 5169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:23:05.586601 5169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "18edfbfb048e4b94892801df59ea2da7"
format_stamp: "Formatted at 2026-08-10 03:23:05 on dist-test-slave-btw8"
server_key: "5700d0f985c774fe9612a0962a5e45bd"
server_key_iv: "d4fc97956d41d2870ccfeb8ddaceca0f"
server_key_version: "encryptionkey@0"
I20260810 03:23:05.586858 5169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:05.634723 5169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:05.640066 5169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:05.640368 5169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:05.641557 5169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:05.643045 5169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:05.643127 5169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:05.643221 5169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:05.643268 5169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:05.680073 5169 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:33771
I20260810 03:23:05.680184 5313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:33771 every 8 connection(s)
I20260810 03:23:05.681622 5169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:23:05.692209 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 5169
I20260810 03:23:05.692476 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:23:05.693295 20443 external_mini_cluster.cc:1428] Setting key 7d2afad3afed5ed4bc388abc00746f97
Aug 10 03:23:05 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332185, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:23:05.706336 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:05.684908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:43815 (local address 127.19.246.254:36501)
0810 03:23:05.685438 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:05.685444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:05.686583 (+ 1139us) server_negotiation.cc:407] Connection header received
0810 03:23:05.688936 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:05.688943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:05.689077 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:05.689230 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:05.691122 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:05.692364 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:05.694259 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:05.694813 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:05.694947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:05.699167 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:05.699174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:05.699178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:05.699231 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:05.702498 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:05.703511 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:05.703519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:05.703528 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:05.703685 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:05.704241 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:05.704245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:05.704248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:05.704559 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:05.704747 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:05.705644 (+ 897us) server_negotiation.cc:299] Negotiation successful
0810 03:23:05.705927 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":267,"threads_started":1}
I20260810 03:23:05.707508 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:05.685489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:43815)
0810 03:23:05.686351 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:05.686419 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:05.688577 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:05.689508 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:05.689519 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:05.690149 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:23:05.690891 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:05.690912 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:05.692609 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:05.692614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:05.694049 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:05.694063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:05.695070 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:05.695075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:05.695470 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:05.696299 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:23:05.696327 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:23:05.698782 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:23:05.702732 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:05.702744 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:05.702774 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:05.703318 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:05.703830 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:05.703836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:05.703840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:05.704089 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:05.704717 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:23:05.704727 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:23:05.705164 (+ 437us) client_negotiation.cc:769] Sending connection context
0810 03:23:05.705474 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 03:23:05.706126 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":334,"threads_started":1}
I20260810 03:23:05.709404 5314 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36501
I20260810 03:23:05.709798 5314 heartbeater.cc:464] Registering TS with master...
I20260810 03:23:05.710686 5314 heartbeater.cc:511] Master 127.19.246.254:36501 requested a full tablet report, sending...
I20260810 03:23:05.713227 5099 ts_manager.cc:194] Registered new tserver with Master: 18edfbfb048e4b94892801df59ea2da7 (127.19.246.193:33771)
I20260810 03:23:05.714774 5099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:43815
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:23:05.763744 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36501
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:23:06.146751 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:06.147083 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:06.147193 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:06.154982 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:06.155125 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:06.155257 5321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:23:06.165668 5321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:06.167076 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:06.168510 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:06.183794 5327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:06.183835 5331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:06.183861 5328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:06.184743 5321 server_base.cc:1034] running on GCE node
I20260810 03:23:06.185513 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:06.186653 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:06.187932 5321 hybrid_clock.cc:648] HybridClock initialized: now 1786332186187803 us; error 74 us; skew 500 ppm
I20260810 03:23:06.188354 5321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:06 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332186, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:06.192024 5321 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:23:06.193944 5321 webserver.cc:460] Webserver started at http://127.19.246.194:45451/ using document root <none> and password file <none>
I20260810 03:23:06.194751 5321 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:06.194916 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:06.195266 5321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:23:06.198513 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "c15be7827cc2401d8b6dfbfa7021c3a7"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "f987734935419023d215659c57f4213d"
server_key_iv: "88554428522df38826e631e158102c44"
server_key_version: "encryptionkey@0"
I20260810 03:23:06.199455 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "c15be7827cc2401d8b6dfbfa7021c3a7"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "f987734935419023d215659c57f4213d"
server_key_iv: "88554428522df38826e631e158102c44"
server_key_version: "encryptionkey@0"
I20260810 03:23:06.205343 5321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:23:06.209662 5338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:06.211476 5321 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:23:06.211772 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "c15be7827cc2401d8b6dfbfa7021c3a7"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "f987734935419023d215659c57f4213d"
server_key_iv: "88554428522df38826e631e158102c44"
server_key_version: "encryptionkey@0"
I20260810 03:23:06.212049 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:06.248293 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:06.253607 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:06.253980 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:06.255133 5321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:06.256732 5321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:06.256844 5321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:06.257016 5321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:06.257098 5321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:06.295833 5321 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:36433
I20260810 03:23:06.295892 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:36433 every 8 connection(s)
I20260810 03:23:06.297272 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:23:06.300211 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 5321
I20260810 03:23:06.300580 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:23:06.301345 20443 external_mini_cluster.cc:1428] Setting key d3ad59631f6bba09f83f4fb67dde0b17
Aug 10 03:23:06 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332186, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:23:06.324005 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:06.300408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:47143 (local address 127.19.246.254:36501)
0810 03:23:06.301009 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:06.301018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:06.302523 (+ 1505us) server_negotiation.cc:407] Connection header received
0810 03:23:06.305525 (+ 3002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:06.305531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:06.305692 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:06.305838 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:06.307897 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:06.308727 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:06.310716 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:06.311230 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:06.311434 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:06.315768 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:06.315778 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:06.315785 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:06.315842 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:06.320115 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:06.321160 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:06.321166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:06.321176 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:06.321330 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:06.321852 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:06.321857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:06.321860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:06.322210 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:06.322409 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:06.323248 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 03:23:06.323589 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":295,"threads_started":1}
I20260810 03:23:06.325148 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:06.301219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:47143)
0810 03:23:06.302240 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:06.302307 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:06.305144 (+ 2837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:06.306159 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:06.306169 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:06.306794 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:23:06.307658 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:06.307689 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:06.308951 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:06.308955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:06.310366 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:06.310383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:06.311562 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:06.311570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:06.311961 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:06.312796 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:23:06.312826 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:23:06.315431 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:23:06.320358 (+ 4927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:06.320370 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:06.320396 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:06.320968 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:06.321462 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:06.321468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:06.321471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:06.321707 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:06.322370 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:23:06.322379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:23:06.322806 (+ 427us) client_negotiation.cc:769] Sending connection context
0810 03:23:06.323140 (+ 334us) client_negotiation.cc:240] Negotiation successful
0810 03:23:06.323830 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":462,"threads_started":1}
I20260810 03:23:06.327003 5459 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36501
I20260810 03:23:06.327417 5459 heartbeater.cc:464] Registering TS with master...
I20260810 03:23:06.328531 5459 heartbeater.cc:511] Master 127.19.246.254:36501 requested a full tablet report, sending...
I20260810 03:23:06.330586 5099 ts_manager.cc:194] Registered new tserver with Master: c15be7827cc2401d8b6dfbfa7021c3a7 (127.19.246.194:36433)
I20260810 03:23:06.332015 5099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:47143
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:23:06.380525 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36501
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:23:06.718346 5314 heartbeater.cc:503] Master 127.19.246.254:36501 was elected leader, sending a full tablet report...
W20260810 03:23:06.790675 5465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:06.791078 5465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:06.791307 5465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:06.799566 5465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:06.799754 5465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:06.799952 5465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:23:06.810719 5465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:06.812288 5465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:06.813946 5465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:06.834728 5470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:06.834707 5471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:06.835315 5473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:06.837244 5465 server_base.cc:1034] running on GCE node
I20260810 03:23:06.838250 5465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:06.839838 5465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:06.841172 5465 hybrid_clock.cc:648] HybridClock initialized: now 1786332186841112 us; error 51 us; skew 500 ppm
I20260810 03:23:06.841720 5465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:06 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332186, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:06.846294 5465 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:23:06.848513 5465 webserver.cc:460] Webserver started at http://127.19.246.195:42391/ using document root <none> and password file <none>
I20260810 03:23:06.849382 5465 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:06.849565 5465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:06.849937 5465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:23:06.853683 5465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "45d31b64fec6499fb53242bed5364223"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "054ff97e83387faa38d14f62bb9fedf3"
server_key_iv: "ca7590cbb44c1a326ffab8ceafd1d8d0"
server_key_version: "encryptionkey@0"
I20260810 03:23:06.854717 5465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "45d31b64fec6499fb53242bed5364223"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "054ff97e83387faa38d14f62bb9fedf3"
server_key_iv: "ca7590cbb44c1a326ffab8ceafd1d8d0"
server_key_version: "encryptionkey@0"
I20260810 03:23:06.861922 5465 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260810 03:23:06.866436 5480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:06.868592 5465 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 03:23:06.868844 5465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "45d31b64fec6499fb53242bed5364223"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "054ff97e83387faa38d14f62bb9fedf3"
server_key_iv: "ca7590cbb44c1a326ffab8ceafd1d8d0"
server_key_version: "encryptionkey@0"
I20260810 03:23:06.869124 5465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:06.887853 5465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:06.895105 5465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:06.895577 5465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:06.897123 5465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:06.899641 5465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:06.899773 5465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:06.899933 5465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:06.900015 5465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:06.948788 5465 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:36135
I20260810 03:23:06.948822 5604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:36135 every 8 connection(s)
I20260810 03:23:06.951016 5465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:23:06.960671 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 5465
I20260810 03:23:06.960945 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:23:06.961512 20443 external_mini_cluster.cc:1428] Setting key 2f65d354a912558012fb654891b5c7d9
Aug 10 03:23:06 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332186, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:23:06.981834 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:06.954306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:42697 (local address 127.19.246.254:36501)
0810 03:23:06.954934 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:06.954945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:06.956138 (+ 1193us) server_negotiation.cc:407] Connection header received
0810 03:23:06.959755 (+ 3617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:06.959761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:06.959866 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:06.959987 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:06.962743 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:06.964369 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:06.966521 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:06.967520 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:06.967698 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:06.972296 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:06.972319 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:06.972332 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:06.972403 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:06.977047 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:06.978077 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:06.978081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:06.978083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:06.978241 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:06.978953 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:06.978962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:06.978968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:06.979492 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:06.979727 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:06.980855 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0810 03:23:06.981193 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":255,"threads_started":1}
I20260810 03:23:06.982508 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:06.954846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:42697)
0810 03:23:06.955898 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:06.955971 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:06.959317 (+ 3346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:06.960390 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:06.960409 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:06.961415 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:23:06.962456 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:06.962489 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:06.964578 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:06.964584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:06.966324 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:06.966340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:06.967812 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:06.967818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:06.968049 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:06.969264 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:23:06.969296 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:23:06.971947 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:23:06.977245 (+ 5298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:06.977257 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:06.977280 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:06.977912 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:06.978415 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:06.978424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:06.978431 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:06.978766 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:06.979698 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:23:06.979706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:23:06.980280 (+ 574us) client_negotiation.cc:769] Sending connection context
0810 03:23:06.980680 (+ 400us) client_negotiation.cc:240] Negotiation successful
0810 03:23:06.981165 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":430,"threads_started":1}
I20260810 03:23:06.984140 5605 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36501
I20260810 03:23:06.984483 5605 heartbeater.cc:464] Registering TS with master...
I20260810 03:23:06.985241 5605 heartbeater.cc:511] Master 127.19.246.254:36501 requested a full tablet report, sending...
I20260810 03:23:06.986897 5099 ts_manager.cc:194] Registered new tserver with Master: 45d31b64fec6499fb53242bed5364223 (127.19.246.195:36135)
I20260810 03:23:06.987905 5099 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:42697
I20260810 03:23:06.992172 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:23:07.016429 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:06.997792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.19.246.254:36501)
0810 03:23:06.998067 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:06.998075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:06.998559 (+ 484us) server_negotiation.cc:407] Connection header received
0810 03:23:06.998786 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:06.998791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:06.998885 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:06.998979 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:07.001450 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.002249 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.005023 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.005467 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.005591 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.009517 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:07.009523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:07.009526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:07.009570 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:07.012621 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:07.013820 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:07.013825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:07.013827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:07.013991 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:07.014597 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:07.014603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:07.014605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:07.014985 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:07.015230 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.015835 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.016140 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:23:07.021660 5099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:23:07.030177 5099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:23:07.065649 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.045903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33771 (local address 127.0.0.1:39942)
0810 03:23:07.047068 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:07.047108 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:07.047313 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:07.048745 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:07.048762 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:07.048801 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:07.049353 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.049374 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.056534 (+ 7160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.056545 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.061587 (+ 5042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.061608 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.064034 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.064043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.064448 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.064542 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 03:23:07.064678 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 03:23:07.065106 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":1,"thread_start_us":752,"threads_started":1}
I20260810 03:23:07.066674 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.046541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.19.246.193:33771)
0810 03:23:07.047981 (+ 1440us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.047996 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.048034 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:23:07.048151 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.048161 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.048446 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.051377 (+ 2931us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:07.051963 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.056268 (+ 4305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.061862 (+ 5594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.063780 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.064023 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.066053 (+ 2030us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.066147 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.066343 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1232,"thread_start_us":1029,"threads_started":1}
I20260810 03:23:07.069211 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.048917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.19.246.194:36433)
0810 03:23:07.059876 (+ 10959us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.059890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.060861 (+ 971us) server_negotiation.cc:407] Connection header received
0810 03:23:07.060985 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.060993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.061319 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.061581 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:07.062575 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.063989 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.066988 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.068076 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.068264 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.068450 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.068713 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.068892 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10739,"thread_start_us":10975,"threads_started":1}
I20260810 03:23:07.069265 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.048377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:36433 (local address 127.0.0.1:47940)
0810 03:23:07.060547 (+ 12170us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:07.060568 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:07.060716 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:07.061601 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:07.061607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:07.061642 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:07.062355 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.062373 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.064306 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.064317 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.066737 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.066754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.068311 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.068318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.068504 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.068539 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:23:07.068666 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 03:23:07.068813 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12022,"mutex_wait_us":36,"spinlock_wait_cycles":16256,"thread_start_us":11804,"threads_started":1}
I20260810 03:23:07.071266 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.047634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36135 (local address 127.0.0.1:48300)
0810 03:23:07.048273 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:07.048293 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:07.048418 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:07.061705 (+ 13287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:07.061716 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:07.061745 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:07.062435 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.062455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.064708 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.064713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.068312 (+ 3599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.068328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.070292 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.070303 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.070498 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.070540 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:23:07.070643 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 03:23:07.070782 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":63,"thread_start_us":339,"threads_started":1}
I20260810 03:23:07.071179 5246 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd30db0293847408a40ce98562b8fff (DEFAULT_TABLE table=test-table [id=2eac54d8d8cf46b49827446410681a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:23:07.072309 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.047975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.19.246.195:36135)
0810 03:23:07.061102 (+ 13127us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.061110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.061142 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:23:07.061231 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.061237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.061425 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.062112 (+ 687us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:07.062700 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.064535 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.068543 (+ 4008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.070067 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.070291 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.070464 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.071595 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.071796 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12940,"thread_start_us":12756,"threads_started":1}
I20260810 03:23:07.072955 5246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd30db0293847408a40ce98562b8fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:23:07.073681 5390 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd30db0293847408a40ce98562b8fff (DEFAULT_TABLE table=test-table [id=2eac54d8d8cf46b49827446410681a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:23:07.075194 5390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd30db0293847408a40ce98562b8fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:23:07.075968 5535 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd30db0293847408a40ce98562b8fff (DEFAULT_TABLE table=test-table [id=2eac54d8d8cf46b49827446410681a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:23:07.077198 5535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd30db0293847408a40ce98562b8fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:23:07.089458 5624 tablet_bootstrap.cc:492] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Bootstrap starting.
I20260810 03:23:07.090427 5625 tablet_bootstrap.cc:492] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: Bootstrap starting.
I20260810 03:23:07.093035 5626 tablet_bootstrap.cc:492] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Bootstrap starting.
I20260810 03:23:07.094497 5624 tablet_bootstrap.cc:654] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Neither blocks nor log segments found. Creating new log.
I20260810 03:23:07.094512 5625 tablet_bootstrap.cc:654] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: Neither blocks nor log segments found. Creating new log.
I20260810 03:23:07.095795 5624 log.cc:824] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:23:07.095887 5625 log.cc:824] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:23:07.097151 5626 tablet_bootstrap.cc:654] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Neither blocks nor log segments found. Creating new log.
I20260810 03:23:07.098193 5626 log.cc:824] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Log is configured to *not* fsync() on all Append() calls
I20260810 03:23:07.099097 5624 tablet_bootstrap.cc:492] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: No bootstrap required, opened a new log
I20260810 03:23:07.099442 5624 ts_tablet_manager.cc:1397] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260810 03:23:07.101157 5625 tablet_bootstrap.cc:492] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: No bootstrap required, opened a new log
I20260810 03:23:07.101208 5626 tablet_bootstrap.cc:492] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: No bootstrap required, opened a new log
I20260810 03:23:07.101442 5625 ts_tablet_manager.cc:1397] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 03:23:07.101500 5626 ts_tablet_manager.cc:1397] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260810 03:23:07.108826 5626 raft_consensus.cc:348] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.109278 5626 raft_consensus.cc:374] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:23:07.109375 5626 raft_consensus.cc:729] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d31b64fec6499fb53242bed5364223, State: Initialized, Role: FOLLOWER
I20260810 03:23:07.110102 5626 consensus_queue.cc:260] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.110777 5624 raft_consensus.cc:348] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.111268 5624 raft_consensus.cc:374] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:23:07.111404 5624 raft_consensus.cc:729] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18edfbfb048e4b94892801df59ea2da7, State: Initialized, Role: FOLLOWER
I20260810 03:23:07.111327 5605 heartbeater.cc:503] Master 127.19.246.254:36501 was elected leader, sending a full tablet report...
I20260810 03:23:07.112112 5626 ts_tablet_manager.cc:1428] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260810 03:23:07.112157 5624 consensus_queue.cc:260] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.112442 5625 raft_consensus.cc:348] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.112977 5625 raft_consensus.cc:374] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:23:07.113091 5625 raft_consensus.cc:729] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15be7827cc2401d8b6dfbfa7021c3a7, State: Initialized, Role: FOLLOWER
I20260810 03:23:07.113921 5625 consensus_queue.cc:260] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.115687 5624 ts_tablet_manager.cc:1428] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260810 03:23:07.116803 5459 heartbeater.cc:503] Master 127.19.246.254:36501 was elected leader, sending a full tablet report...
I20260810 03:23:07.117563 5625 ts_tablet_manager.cc:1428] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260810 03:23:07.143631 5631 raft_consensus.cc:482] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:23:07.144049 5631 raft_consensus.cc:504] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.146263 5631 leader_election.cc:288] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45d31b64fec6499fb53242bed5364223 (127.19.246.195:36135), c15be7827cc2401d8b6dfbfa7021c3a7 (127.19.246.194:36433)
I20260810 03:23:07.154659 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.148032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:42957 (local address 127.19.246.194:36433)
0810 03:23:07.148290 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.148298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.148568 (+ 270us) server_negotiation.cc:407] Connection header received
0810 03:23:07.148740 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.148746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.148847 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.148984 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:07.149785 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.150870 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.152768 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.153602 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.153788 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.153875 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.154197 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.154349 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 03:23:07.154738 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.146741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:42093 (local address 127.19.246.195:36135)
0810 03:23:07.147077 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.147086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.147667 (+ 581us) server_negotiation.cc:407] Connection header received
0810 03:23:07.147914 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.147923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.148045 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.148159 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:07.149178 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.150418 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.152765 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.153640 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.153856 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.153949 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.154291 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.154440 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260810 03:23:07.154812 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.147680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:36433 (local address 127.19.246.193:42957)
0810 03:23:07.148450 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:07.148467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:07.148593 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:07.149018 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:07.149022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:07.149105 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:07.149548 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.149563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.151027 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.151033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.152596 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.152610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.153857 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.153864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.154024 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.154049 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:23:07.154169 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 03:23:07.154378 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":37,"thread_start_us":361,"threads_started":1}
I20260810 03:23:07.155155 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.146580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36135 (local address 127.19.246.193:42093)
0810 03:23:07.147476 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:07.147496 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:07.147717 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:07.148331 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:07.148337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:07.148380 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:07.148971 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.148999 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.150581 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.150588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.152566 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:07.152581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.153919 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.153926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:07.154106 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.154131 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:23:07.154247 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 03:23:07.154427 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":4,"spinlock_wait_cycles":32640,"thread_start_us":452,"threads_started":1}
I20260810 03:23:07.156152 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd30db0293847408a40ce98562b8fff" candidate_uuid: "18edfbfb048e4b94892801df59ea2da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" is_pre_election: true
I20260810 03:23:07.156251 5557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd30db0293847408a40ce98562b8fff" candidate_uuid: "18edfbfb048e4b94892801df59ea2da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d31b64fec6499fb53242bed5364223" is_pre_election: true
I20260810 03:23:07.156776 5411 raft_consensus.cc:2454] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18edfbfb048e4b94892801df59ea2da7 in term 0.
I20260810 03:23:07.156873 5557 raft_consensus.cc:2454] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18edfbfb048e4b94892801df59ea2da7 in term 0.
I20260810 03:23:07.157557 5190 leader_election.cc:302] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18edfbfb048e4b94892801df59ea2da7, c15be7827cc2401d8b6dfbfa7021c3a7; no voters:
I20260810 03:23:07.158041 5631 raft_consensus.cc:2781] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:23:07.158283 5631 raft_consensus.cc:482] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:23:07.158418 5631 raft_consensus.cc:3037] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:23:07.160925 5631 raft_consensus.cc:504] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.161634 5631 leader_election.cc:288] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [CANDIDATE]: Term 1 election: Requested vote from peers 45d31b64fec6499fb53242bed5364223 (127.19.246.195:36135), c15be7827cc2401d8b6dfbfa7021c3a7 (127.19.246.194:36433)
I20260810 03:23:07.162447 5557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd30db0293847408a40ce98562b8fff" candidate_uuid: "18edfbfb048e4b94892801df59ea2da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d31b64fec6499fb53242bed5364223"
I20260810 03:23:07.162595 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd30db0293847408a40ce98562b8fff" candidate_uuid: "18edfbfb048e4b94892801df59ea2da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7"
I20260810 03:23:07.162734 5557 raft_consensus.cc:3037] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:23:07.162897 5411 raft_consensus.cc:3037] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:23:07.165311 5411 raft_consensus.cc:2454] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18edfbfb048e4b94892801df59ea2da7 in term 1.
I20260810 03:23:07.165284 5557 raft_consensus.cc:2454] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18edfbfb048e4b94892801df59ea2da7 in term 1.
I20260810 03:23:07.165860 5190 leader_election.cc:302] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18edfbfb048e4b94892801df59ea2da7, c15be7827cc2401d8b6dfbfa7021c3a7; no voters:
I20260810 03:23:07.166275 5631 raft_consensus.cc:2781] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:23:07.166836 5631 raft_consensus.cc:686] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 1 LEADER]: Becoming Leader. State: Replica: 18edfbfb048e4b94892801df59ea2da7, State: Running, Role: LEADER
I20260810 03:23:07.167459 5631 consensus_queue.cc:237] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } }
I20260810 03:23:07.172147 5098 catalog_manager.cc:5495] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18edfbfb048e4b94892801df59ea2da7 (127.19.246.193). New cstate: current_term: 1 leader_uuid: "18edfbfb048e4b94892801df59ea2da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } health_report { overall_health: UNKNOWN } } }
W20260810 03:23:07.184744 5315 tablet.cc:2369] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:23:07.203152 5606 tablet.cc:2369] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:23:07.220638 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.213983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.19.246.193:33771)
0810 03:23:07.214211 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.214215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.214296 (+ 81us) server_negotiation.cc:407] Connection header received
0810 03:23:07.214507 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.214511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.214594 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.214787 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:23:07.215149 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.216060 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.217692 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.218175 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.218295 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.218912 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:23:07.219635 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:23:07.219768 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.220006 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.220096 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 03:23:07.230684 5557 raft_consensus.cc:1261] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 1 FOLLOWER]: Refusing update from remote peer 18edfbfb048e4b94892801df59ea2da7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:23:07.230680 5411 raft_consensus.cc:1261] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 1 FOLLOWER]: Refusing update from remote peer 18edfbfb048e4b94892801df59ea2da7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:23:07.231966 5635 consensus_queue.cc:1035] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:23:07.232458 5631 consensus_queue.cc:1035] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:23:07.250372 5639 mvcc.cc:205] Tried to move back new op lower bound from 7316816638885404672 to 7316816638641770496. Current Snapshot: MvccSnapshot[applied={T|T < 7316816638885404672}]
I20260810 03:23:07.264961 5098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:23:07.265349 5098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:23:07.269851 5098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Sending DeleteTablet for 3 replicas of tablet 0dd30db0293847408a40ce98562b8fff
I20260810 03:23:07.271385 5246 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd30db0293847408a40ce98562b8fff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:23:07 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:39942
I20260810 03:23:07.271507 5535 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd30db0293847408a40ce98562b8fff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:23:07 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:48300
I20260810 03:23:07.271744 5390 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd30db0293847408a40ce98562b8fff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:23:07 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:47940
I20260810 03:23:07.272960 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 5169
I20260810 03:23:07.283059 5650 tablet_replica.cc:330] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: stopping tablet replica
I20260810 03:23:07.283784 5650 raft_consensus.cc:2229] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:23:07.284579 5650 raft_consensus.cc:2258] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:23:07.287626 5650 ts_tablet_manager.cc:1905] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:23:07.291384 5651 tablet_replica.cc:330] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: stopping tablet replica
I20260810 03:23:07.291504 5649 tablet_replica.cc:330] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: stopping tablet replica
I20260810 03:23:07.292316 5651 raft_consensus.cc:2229] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:23:07.292940 5649 raft_consensus.cc:2229] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:23:07.293732 5651 raft_consensus.cc:2258] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:23:07.294711 5649 raft_consensus.cc:2258] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:23:07.296545 5651 ts_tablet_manager.cc:1905] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:23:07.300997 5650 ts_tablet_manager.cc:1918] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:23:07.301221 5650 log.cc:1196] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/0dd30db0293847408a40ce98562b8fff
I20260810 03:23:07.301898 5650 ts_tablet_manager.cc:1939] T 0dd30db0293847408a40ce98562b8fff P 45d31b64fec6499fb53242bed5364223: Deleting consensus metadata
I20260810 03:23:07.302767 5649 ts_tablet_manager.cc:1905] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:23:07.304328 5082 catalog_manager.cc:4841] TS 45d31b64fec6499fb53242bed5364223 (127.19.246.195:36135): tablet 0dd30db0293847408a40ce98562b8fff (table test-table [id=2eac54d8d8cf46b49827446410681a97]) successfully deleted
Aug 10 03:23:07 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:23:07.316685 5651 ts_tablet_manager.cc:1918] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:23:07.316919 5651 log.cc:1196] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/0dd30db0293847408a40ce98562b8fff
I20260810 03:23:07.317699 5651 ts_tablet_manager.cc:1939] T 0dd30db0293847408a40ce98562b8fff P c15be7827cc2401d8b6dfbfa7021c3a7: Deleting consensus metadata
I20260810 03:23:07.320029 5082 catalog_manager.cc:4841] TS c15be7827cc2401d8b6dfbfa7021c3a7 (127.19.246.194:36433): tablet 0dd30db0293847408a40ce98562b8fff (table test-table [id=2eac54d8d8cf46b49827446410681a97]) successfully deleted
I20260810 03:23:07.321913 5649 ts_tablet_manager.cc:1918] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:23:07.326004 5649 log.cc:1196] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/0dd30db0293847408a40ce98562b8fff
I20260810 03:23:07.326951 5649 ts_tablet_manager.cc:1939] T 0dd30db0293847408a40ce98562b8fff P 18edfbfb048e4b94892801df59ea2da7: Deleting consensus metadata
I20260810 03:23:07.329329 5082 catalog_manager.cc:4841] TS 18edfbfb048e4b94892801df59ea2da7 (127.19.246.193:33771): tablet 0dd30db0293847408a40ce98562b8fff (table test-table [id=2eac54d8d8cf46b49827446410681a97]) successfully deleted
I20260810 03:23:07.329857 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.274843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.19.246.193:33771)
0810 03:23:07.275112 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.275121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.291480 (+ 16359us) server_negotiation.cc:407] Connection header received
0810 03:23:07.291902 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.291910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.292037 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.292150 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:07.305227 (+ 13077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.306768 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.309652 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.310689 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.310872 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.320566 (+ 9694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:07.320591 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:07.320612 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:07.320674 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:07.325723 (+ 5049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:07.326967 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:07.326976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:07.326979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:07.327294 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:07.327985 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:07.327990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:07.327992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:07.328464 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:07.328807 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.329218 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.329518 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 03:23:07.331177 5309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:07.438331 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 5169
I20260810 03:23:07.461251 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 5321
Aug 10 03:23:07 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:23:07.479229 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.462099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.19.246.194:36433)
0810 03:23:07.462322 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.462327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.462347 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:23:07.462411 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.462415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.462491 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.462636 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:07.464078 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.464928 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.466431 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.466947 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.467085 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.472468 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:07.472488 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:07.472505 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:07.472556 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:07.476098 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:07.476946 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:07.476953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:07.476955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:07.477088 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:07.477485 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:07.477489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:07.477491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:07.477986 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:07.478249 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.478632 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.478866 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 03:23:07.480144 5454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:07.576363 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 5321
I20260810 03:23:07.596796 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 5465
Aug 10 03:23:07 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:23:07.615175 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:07.597758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.19.246.195:36135)
0810 03:23:07.597955 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:07.597960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:07.597981 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:23:07.598059 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:07.598063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:07.598126 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:07.598230 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:07.599835 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.600760 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.602192 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:07.602619 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:07.602782 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:07.608248 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:07.608271 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:07.608289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:07.608368 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:07.612073 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:07.612979 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:07.612988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:07.612990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:07.613170 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:07.613543 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:07.613548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:07.613551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:07.613984 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:07.614263 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:07.614547 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 03:23:07.614760 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:23:07.615964 5600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:07.709735 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 5465
I20260810 03:23:07.731426 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 5065
I20260810 03:23:07.732764 5136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:07.851320 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 5065
I20260810 03:23:27.877588 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36501
--webserver_interface=127.19.246.254
--webserver_port=40991
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:23:28.270424 5657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:28.270802 5657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:28.270978 5657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:28.279172 5657 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:23:28.279299 5657 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:23:28.279340 5657 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:23:28.279470 5657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:23:28.279580 5657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:28.279675 5657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:28.279734 5657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:23:28.279816 5657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:23:28.290509 5657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:28.292045 5657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:28.293478 5657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:28.322264 5664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:28.322507 5668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:28.322257 5665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:28.323515 5657 server_base.cc:1034] running on GCE node
I20260810 03:23:28.324584 5657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:28.326508 5657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:28.327870 5657 hybrid_clock.cc:648] HybridClock initialized: now 1786332208327751 us; error 63 us; skew 500 ppm
I20260810 03:23:28.328318 5657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:28 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332208, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:28.337497 5657 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:23:28.339745 5657 webserver.cc:460] Webserver started at http://127.19.246.254:40991/ using document root <none> and password file <none>
I20260810 03:23:28.340695 5657 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:28.340860 5657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:28.348032 5657 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.007s
I20260810 03:23:28.351908 5676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:28.353801 5657 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:23:28.354038 5657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "52e84229034a4933bc0c2854b206f8c8"
format_stamp: "Formatted at 2026-08-10 03:23:04 on dist-test-slave-btw8"
server_key: "7d70dd2e6f7f3d08880bc5450234b576"
server_key_iv: "3b1dd65161dfedcd49955e40539c46a6"
server_key_version: "encryptionkey@0"
I20260810 03:23:28.355237 5657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:28.379511 5657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:28.385650 5657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:28.386049 5657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:28.410693 5657 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:36501
I20260810 03:23:28.410782 5731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:36501 every 8 connection(s)
I20260810 03:23:28.412820 5657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:23:28.416473 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 5657
I20260810 03:23:28.417409 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:33771
--local_ip_for_outbound_sockets=127.19.246.193
--tserver_master_addrs=127.19.246.254:36501
--webserver_port=40655
--webserver_interface=127.19.246.193
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:23:28.421804 5732 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:23:28.426003 5732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Bootstrap starting.
I20260810 03:23:28.448326 5732 log.cc:824] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:23:28.462793 5732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:23:28.463671 5732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Bootstrap complete.
I20260810 03:23:28.474813 5732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:28.475813 5732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e84229034a4933bc0c2854b206f8c8, State: Initialized, Role: FOLLOWER
I20260810 03:23:28.476646 5732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:28.476924 5732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:23:28.477053 5732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:23:28.477262 5732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:23:28.482662 5732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:28.483428 5732 leader_election.cc:302] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e84229034a4933bc0c2854b206f8c8; no voters:
I20260810 03:23:28.484170 5732 leader_election.cc:288] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:23:28.484344 5736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:23:28.485718 5736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 2 LEADER]: Becoming Leader. State: Replica: 52e84229034a4933bc0c2854b206f8c8, State: Running, Role: LEADER
I20260810 03:23:28.486408 5736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:28.486918 5732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:23:28.491478 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "52e84229034a4933bc0c2854b206f8c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } } }
I20260810 03:23:28.492142 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:23:28.491448 5738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e84229034a4933bc0c2854b206f8c8. Latest consensus state: current_term: 2 leader_uuid: "52e84229034a4933bc0c2854b206f8c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } } }
I20260810 03:23:28.492700 5738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:23:28.506017 5745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:23:28.516247 5745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:23:28.518122 5745 catalog_manager.cc:1259] Loaded cluster ID: 61871238f04346a298cecd6283bb0c12
I20260810 03:23:28.518303 5745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:23:28.525085 5745 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:23:28.527235 5745 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Loaded TSK: 0
I20260810 03:23:28.558315 5745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Generated new TSK 1
I20260810 03:23:28.558847 5745 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:23:28.853386 5734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:28.853811 5734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:28.854017 5734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:28.863327 5734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:28.863566 5734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:28.863790 5734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:23:28.876128 5734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:33771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=40655
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:28.877734 5734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:28.879256 5734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:28.897723 5761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:28.897833 5762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:28.897698 5764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:28.898763 5734 server_base.cc:1034] running on GCE node
I20260810 03:23:28.899595 5734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:28.900820 5734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:28.902061 5734 hybrid_clock.cc:648] HybridClock initialized: now 1786332208901989 us; error 65 us; skew 500 ppm
I20260810 03:23:28.902428 5734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:28 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332208, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:28.906603 5734 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:23:28.908931 5734 webserver.cc:460] Webserver started at http://127.19.246.193:40655/ using document root <none> and password file <none>
I20260810 03:23:28.910068 5734 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:28.910265 5734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:28.918758 5734 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:23:28.922947 5771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:28.924994 5734 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 03:23:28.925208 5734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "18edfbfb048e4b94892801df59ea2da7"
format_stamp: "Formatted at 2026-08-10 03:23:05 on dist-test-slave-btw8"
server_key: "5700d0f985c774fe9612a0962a5e45bd"
server_key_iv: "d4fc97956d41d2870ccfeb8ddaceca0f"
server_key_version: "encryptionkey@0"
I20260810 03:23:28.926695 5734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:28.951216 5734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:28.957615 5734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:28.957959 5734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:28.959270 5734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:28.961345 5734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:28.961470 5734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:28.961566 5734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:28.961652 5734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:29.015156 5734 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:33771
I20260810 03:23:29.015269 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:33771 every 8 connection(s)
I20260810 03:23:29.016875 5734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:23:29.024740 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 5734
I20260810 03:23:29.025439 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:36433
--local_ip_for_outbound_sockets=127.19.246.194
--tserver_master_addrs=127.19.246.254:36501
--webserver_port=45451
--webserver_interface=127.19.246.194
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:23:29 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332208, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:23:29.044590 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:29.020138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:55589 (local address 127.19.246.254:36501)
0810 03:23:29.020785 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:29.020805 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:29.021424 (+ 619us) server_negotiation.cc:407] Connection header received
0810 03:23:29.023775 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:29.023787 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:29.024141 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:29.024540 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:29.026255 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:29.027598 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:29.029457 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:29.030122 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:29.030353 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:29.034991 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:29.035006 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:29.035030 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:29.035096 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:29.038900 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:29.040206 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:29.040216 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:29.040222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:29.040438 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:29.041020 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:29.041024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:29.041027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:29.041602 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:29.041848 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:29.042962 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0810 03:23:29.043455 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":284,"threads_started":1}
I20260810 03:23:29.044981 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:29.020399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:55589)
0810 03:23:29.021222 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:29.021262 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:29.023122 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:29.024733 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:29.024748 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:29.025383 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:23:29.026026 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:29.026045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:29.027811 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:29.027816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:29.029290 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:29.029306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:29.030413 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:29.030421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:29.030725 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:29.031534 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:23:29.031560 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:23:29.034595 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:23:29.039147 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:29.039160 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:29.039180 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:29.040002 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:29.040586 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:29.040595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:29.040602 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:29.040887 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:29.041840 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:23:29.041852 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:23:29.042338 (+ 486us) client_negotiation.cc:769] Sending connection context
0810 03:23:29.042695 (+ 357us) client_negotiation.cc:240] Negotiation successful
0810 03:23:29.043414 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":348,"threads_started":1}
I20260810 03:23:29.048290 5890 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36501
I20260810 03:23:29.048719 5890 heartbeater.cc:464] Registering TS with master...
I20260810 03:23:29.049540 5890 heartbeater.cc:511] Master 127.19.246.254:36501 requested a full tablet report, sending...
I20260810 03:23:29.052110 5694 ts_manager.cc:194] Registered new tserver with Master: 18edfbfb048e4b94892801df59ea2da7 (127.19.246.193:33771)
I20260810 03:23:29.053756 5694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:55589
W20260810 03:23:29.404577 5894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:29.404965 5894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:29.405154 5894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:29.413719 5894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:29.413909 5894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:29.414109 5894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:23:29.425107 5894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:36433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=45451
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:29.426594 5894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:29.428139 5894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:29.443533 5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:29.444191 5904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:29.444224 5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:29.444741 5894 server_base.cc:1034] running on GCE node
I20260810 03:23:29.445365 5894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:29.446367 5894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:29.447649 5894 hybrid_clock.cc:648] HybridClock initialized: now 1786332209447578 us; error 62 us; skew 500 ppm
I20260810 03:23:29.448043 5894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:29 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332209, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:29.451713 5894 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:23:29.453367 5894 webserver.cc:460] Webserver started at http://127.19.246.194:45451/ using document root <none> and password file <none>
I20260810 03:23:29.454228 5894 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:29.454362 5894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:29.460558 5894 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:23:29.463951 5911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:29.465662 5894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:23:29.465906 5894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "c15be7827cc2401d8b6dfbfa7021c3a7"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "f987734935419023d215659c57f4213d"
server_key_iv: "88554428522df38826e631e158102c44"
server_key_version: "encryptionkey@0"
I20260810 03:23:29.467123 5894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:29.498322 5894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:29.504024 5894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:29.504401 5894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:29.505867 5894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:29.507737 5894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:29.507853 5894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:29.508003 5894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:29.508097 5894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:29.547150 5894 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:36433
I20260810 03:23:29.547269 6031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:36433 every 8 connection(s)
I20260810 03:23:29.548815 5894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:23:29.554577 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 5894
I20260810 03:23:29.555415 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:36135
--local_ip_for_outbound_sockets=127.19.246.195
--tserver_master_addrs=127.19.246.254:36501
--webserver_port=42391
--webserver_interface=127.19.246.195
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:23:29 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332209, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:23:29.576135 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:29.551668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:34695 (local address 127.19.246.254:36501)
0810 03:23:29.552382 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:29.552392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:29.553061 (+ 669us) server_negotiation.cc:407] Connection header received
0810 03:23:29.555800 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:29.555805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:29.555885 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:29.555990 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:29.558424 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:29.559481 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:29.562144 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:29.562631 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:29.562739 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:29.567821 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:29.567827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:29.567830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:29.567870 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:29.571809 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:29.572960 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:29.572970 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:29.572976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:29.573158 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:29.573752 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:29.573760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:29.573766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:29.574244 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:29.574411 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:29.575245 (+ 834us) server_negotiation.cc:299] Negotiation successful
0810 03:23:29.575609 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":285,"threads_started":1}
I20260810 03:23:29.576960 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:29.552080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:34695)
0810 03:23:29.552887 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:29.552928 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:29.555344 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:29.556314 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:29.556331 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:29.557232 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:23:29.558153 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:29.558183 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:29.559643 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:29.559651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:29.561944 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:29.561964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:29.562900 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:29.562910 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:29.563271 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:29.564166 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:23:29.564203 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:23:29.567475 (+ 3272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:23:29.572069 (+ 4594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:29.572080 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:29.572097 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:29.572775 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:29.573306 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:29.573310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:29.573313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:29.573557 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:29.574399 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:23:29.574406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:23:29.574834 (+ 428us) client_negotiation.cc:769] Sending connection context
0810 03:23:29.575109 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 03:23:29.575702 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":330,"threads_started":1}
I20260810 03:23:29.578802 6032 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36501
I20260810 03:23:29.579227 6032 heartbeater.cc:464] Registering TS with master...
I20260810 03:23:29.580518 6032 heartbeater.cc:511] Master 127.19.246.254:36501 requested a full tablet report, sending...
I20260810 03:23:29.582737 5694 ts_manager.cc:194] Registered new tserver with Master: c15be7827cc2401d8b6dfbfa7021c3a7 (127.19.246.194:36433)
I20260810 03:23:29.583727 5694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:34695
W20260810 03:23:29.947222 6037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:29.947625 6037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:29.947785 6037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:29.956338 6037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:29.956488 6037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:29.956621 6037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:23:29.967314 6037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:36135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=42391
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:29.968820 6037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:29.970275 6037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:29.985522 6042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:29.985704 6045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:29.985953 6043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:29.986493 6037 server_base.cc:1034] running on GCE node
I20260810 03:23:29.987300 6037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:29.988593 6037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:29.989858 6037 hybrid_clock.cc:648] HybridClock initialized: now 1786332209989813 us; error 39 us; skew 500 ppm
I20260810 03:23:29.990222 6037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:29 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332209, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:29.994011 6037 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:23:29.996194 6037 webserver.cc:460] Webserver started at http://127.19.246.195:42391/ using document root <none> and password file <none>
I20260810 03:23:29.997043 6037 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:29.997233 6037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:30.004873 6037 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260810 03:23:30.008529 6053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:30.010483 6037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:23:30.010746 6037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "45d31b64fec6499fb53242bed5364223"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "054ff97e83387faa38d14f62bb9fedf3"
server_key_iv: "ca7590cbb44c1a326ffab8ceafd1d8d0"
server_key_version: "encryptionkey@0"
I20260810 03:23:30.012109 6037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:30.035615 6037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:30.040915 6037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:30.041296 6037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:30.042634 6037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:30.044329 6037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:30.044476 6037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:30.044651 6037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:30.044759 6037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:30.057979 5890 heartbeater.cc:503] Master 127.19.246.254:36501 was elected leader, sending a full tablet report...
I20260810 03:23:30.085054 6037 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:36135
I20260810 03:23:30.085112 6177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:36135 every 8 connection(s)
I20260810 03:23:30.086717 6037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:23:30.087090 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 6037
Aug 10 03:23:30 dist-test-slave-btw8 krb5kdc[5051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332209, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:23:30.105605 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.088775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.19.246.254:36501)
0810 03:23:30.089235 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.089240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.089261 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:23:30.089387 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.089394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.089488 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.089626 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:30.091836 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.092622 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.094343 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.095032 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.095114 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.098900 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:30.098905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:30.098909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:30.098950 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:30.102258 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.103144 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.103151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.103154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.103311 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.103795 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.103801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.103804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.104154 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:30.104516 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.104828 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.105087 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":345,"thread_start_us":250,"threads_started":1}
I20260810 03:23:30.110983 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.090068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:50471 (local address 127.19.246.254:36501)
0810 03:23:30.090610 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.090616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.091861 (+ 1245us) server_negotiation.cc:407] Connection header received
0810 03:23:30.093934 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.093941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.094029 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.094151 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:30.096193 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.097409 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.099294 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.099831 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.099984 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.103848 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:30.103854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:30.103857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:30.103895 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:30.107332 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.108335 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.108341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.108345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.108511 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.109017 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.109022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.109026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.109358 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:30.109525 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.110297 (+ 772us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.110554 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":419,"thread_start_us":261,"threads_started":1}
I20260810 03:23:30.112108 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.090770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:50471)
0810 03:23:30.091697 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:30.091767 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:30.093615 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:30.094385 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:30.094443 (+ 58us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:30.095200 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:23:30.095965 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.095994 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.097632 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.097636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.099059 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.099074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.100098 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.100107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.100414 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.101098 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:23:30.101124 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:23:30.103560 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:23:30.107571 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:30.107584 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:30.107606 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:30.108183 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:30.108664 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:23:30.108668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:23:30.108670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:23:30.108885 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:23:30.109500 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:23:30.109510 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:23:30.109909 (+ 399us) client_negotiation.cc:769] Sending connection context
0810 03:23:30.110195 (+ 286us) client_negotiation.cc:240] Negotiation successful
0810 03:23:30.110796 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":394,"threads_started":1}
I20260810 03:23:30.113713 6178 heartbeater.cc:347] Connected to a master server at 127.19.246.254:36501
I20260810 03:23:30.114138 6178 heartbeater.cc:464] Registering TS with master...
I20260810 03:23:30.115073 6178 heartbeater.cc:511] Master 127.19.246.254:36501 requested a full tablet report, sending...
I20260810 03:23:30.117069 5694 ts_manager.cc:194] Registered new tserver with Master: 45d31b64fec6499fb53242bed5364223 (127.19.246.195:36135)
I20260810 03:23:30.118010 5694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:50471
I20260810 03:23:30.122037 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:23:30.130507 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.123152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.19.246.254:36501)
0810 03:23:30.123410 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.123416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.124172 (+ 756us) server_negotiation.cc:407] Connection header received
0810 03:23:30.124481 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.124487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.124643 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.124843 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:23:30.125381 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.126208 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.128033 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.128529 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.128698 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.129329 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:23:30.129654 (+ 325us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 03:23:30.130151 (+ 497us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54716: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 03:23:30.130714 6182 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54716: token expired
W20260810 03:23:30.131314 5609 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.246.254:36501)
I20260810 03:23:30.146905 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.132325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.19.246.254:36501)
0810 03:23:30.132560 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.132577 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.132604 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:23:30.132728 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.132732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.132801 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.132890 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:30.134446 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.135478 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.137212 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.137638 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.137761 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.141076 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:30.141082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:30.141086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:30.141127 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:30.144138 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.144933 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.144938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.144940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.145082 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.145468 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.145471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.145473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.145815 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:30.146044 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.146376 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.146564 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:23:30.150952 5694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:23:30.158962 5694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:23:30.192737 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.173164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.19.246.193:33771)
0810 03:23:30.174664 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.174676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.174710 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:23:30.174813 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.174821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.175134 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.178187 (+ 3053us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:30.179861 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.184469 (+ 4608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.189477 (+ 5008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.191164 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.191520 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.191739 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.192200 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.192393 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1175,"threads_started":1}
I20260810 03:23:30.192802 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.172950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:33771 (local address 127.0.0.1:38136)
0810 03:23:30.174070 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:30.174111 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:30.174283 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:30.175494 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:30.175506 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:30.175551 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:30.176179 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.176204 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.184791 (+ 8587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.184803 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.188859 (+ 4056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.188902 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.191565 (+ 2663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.191580 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.191891 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.191959 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 03:23:30.192120 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 03:23:30.192368 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":699,"threads_started":1}
I20260810 03:23:30.197901 5821 tablet_service.cc:1467] Processing CreateTablet for tablet 13e664f8a27349fa8f7831538366ed3e (DEFAULT_TABLE table=test-table [id=7584b22d7c9c4d0a9c8620c36e6214e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:23:30.199276 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.174821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.19.246.194:36433)
0810 03:23:30.190124 (+ 15303us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.190135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.190166 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:23:30.190269 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.190276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.190517 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.191091 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:30.191932 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.193429 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.196271 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.197976 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.198194 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.198394 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.198706 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.198908 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15095,"thread_start_us":15899,"threads_started":1}
I20260810 03:23:30.199669 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.174542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:36433 (local address 127.0.0.1:48636)
0810 03:23:30.175170 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:30.175188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:30.175299 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:30.191097 (+ 15798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:30.191102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:30.191137 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:30.191668 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.191682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.193598 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.193608 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.196014 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.196035 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.198256 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.198262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.198438 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.198482 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:23:30.198583 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:23:30.198887 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":4,"thread_start_us":349,"threads_started":1}
I20260810 03:23:30.199713 5821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e664f8a27349fa8f7831538366ed3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:23:30.200217 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.175271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36135 (local address 127.0.0.1:60986)
0810 03:23:30.189315 (+ 14044us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:30.189339 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:30.189491 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:30.190655 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:30.190660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:30.190701 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:30.191081 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.191096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.193079 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.193084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.196569 (+ 3485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.196585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.199221 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.199227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.199487 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.199531 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:23:30.199638 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 03:23:30.199825 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13892,"mutex_wait_us":84,"thread_start_us":13712,"threads_started":1}
I20260810 03:23:30.201032 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.175970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.19.246.195:36135)
0810 03:23:30.188470 (+ 12500us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.188487 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.189641 (+ 1154us) server_negotiation.cc:407] Connection header received
0810 03:23:30.189778 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.189788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.190103 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.190336 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:30.191368 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.192890 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.196795 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.198625 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.198837 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.199009 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.200517 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.200674 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12219,"thread_start_us":12517,"threads_started":1}
I20260810 03:23:30.204249 5962 tablet_service.cc:1467] Processing CreateTablet for tablet 13e664f8a27349fa8f7831538366ed3e (DEFAULT_TABLE table=test-table [id=7584b22d7c9c4d0a9c8620c36e6214e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:23:30.204927 6107 tablet_service.cc:1467] Processing CreateTablet for tablet 13e664f8a27349fa8f7831538366ed3e (DEFAULT_TABLE table=test-table [id=7584b22d7c9c4d0a9c8620c36e6214e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:23:30.205858 5962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e664f8a27349fa8f7831538366ed3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:23:30.206403 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e664f8a27349fa8f7831538366ed3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:23:30.217514 6195 tablet_bootstrap.cc:492] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Bootstrap starting.
I20260810 03:23:30.222760 6197 tablet_bootstrap.cc:492] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Bootstrap starting.
I20260810 03:23:30.223819 6195 tablet_bootstrap.cc:654] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Neither blocks nor log segments found. Creating new log.
I20260810 03:23:30.224992 6195 log.cc:824] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:23:30.227746 6197 tablet_bootstrap.cc:654] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Neither blocks nor log segments found. Creating new log.
I20260810 03:23:30.228875 6195 tablet_bootstrap.cc:492] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: No bootstrap required, opened a new log
I20260810 03:23:30.228883 6197 log.cc:824] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:23:30.229223 6195 ts_tablet_manager.cc:1397] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260810 03:23:30.231384 6198 tablet_bootstrap.cc:492] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Bootstrap starting.
I20260810 03:23:30.232666 6197 tablet_bootstrap.cc:492] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: No bootstrap required, opened a new log
I20260810 03:23:30.233083 6197 ts_tablet_manager.cc:1397] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260810 03:23:30.236354 6198 tablet_bootstrap.cc:654] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Neither blocks nor log segments found. Creating new log.
I20260810 03:23:30.237917 6198 log.cc:824] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Log is configured to *not* fsync() on all Append() calls
I20260810 03:23:30.238587 6195 raft_consensus.cc:348] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.239173 6195 raft_consensus.cc:374] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:23:30.239295 6195 raft_consensus.cc:729] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18edfbfb048e4b94892801df59ea2da7, State: Initialized, Role: FOLLOWER
I20260810 03:23:30.240080 6195 consensus_queue.cc:260] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.241990 6198 tablet_bootstrap.cc:492] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: No bootstrap required, opened a new log
I20260810 03:23:30.242081 6195 ts_tablet_manager.cc:1428] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260810 03:23:30.242321 6198 ts_tablet_manager.cc:1397] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20260810 03:23:30.244398 6197 raft_consensus.cc:348] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.244915 6197 raft_consensus.cc:374] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:23:30.245015 6197 raft_consensus.cc:729] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15be7827cc2401d8b6dfbfa7021c3a7, State: Initialized, Role: FOLLOWER
I20260810 03:23:30.245675 6197 consensus_queue.cc:260] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.247000 6032 heartbeater.cc:503] Master 127.19.246.254:36501 was elected leader, sending a full tablet report...
I20260810 03:23:30.247830 6197 ts_tablet_manager.cc:1428] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260810 03:23:30.253176 6198 raft_consensus.cc:348] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.253648 6198 raft_consensus.cc:374] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:23:30.253770 6198 raft_consensus.cc:729] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d31b64fec6499fb53242bed5364223, State: Initialized, Role: FOLLOWER
I20260810 03:23:30.254423 6198 consensus_queue.cc:260] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.255668 6178 heartbeater.cc:503] Master 127.19.246.254:36501 was elected leader, sending a full tablet report...
I20260810 03:23:30.256343 6198 ts_tablet_manager.cc:1428] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260810 03:23:30.269972 5891 tablet.cc:2369] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:23:30.301436 6033 tablet.cc:2369] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:23:30.338845 6179 tablet.cc:2369] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:23:30.461239 6205 raft_consensus.cc:482] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:23:30.461642 6205 raft_consensus.cc:504] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.463717 6205 leader_election.cc:288] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c15be7827cc2401d8b6dfbfa7021c3a7 (127.19.246.194:36433), 45d31b64fec6499fb53242bed5364223 (127.19.246.195:36135)
I20260810 03:23:30.472150 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.464308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:60373 (local address 127.19.246.194:36433)
0810 03:23:30.464520 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.464525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.465113 (+ 588us) server_negotiation.cc:407] Connection header received
0810 03:23:30.465362 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.465368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.465460 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.465594 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:30.466442 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.467729 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.469858 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.470893 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.471061 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.471122 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.471680 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.471799 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:23:30.472311 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.464193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:36433 (local address 127.19.246.193:60373)
0810 03:23:30.464971 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:30.464992 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:30.465187 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:30.465754 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:30.465759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:30.465797 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:30.466238 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.466251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.467889 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.467894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.469617 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.469644 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.471174 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.471182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.471459 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.471502 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 03:23:30.471633 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:23:30.471820 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":437,"threads_started":1}
I20260810 03:23:30.473125 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.465344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:48099 (local address 127.19.246.195:36135)
0810 03:23:30.465646 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.465654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.465905 (+ 251us) server_negotiation.cc:407] Connection header received
0810 03:23:30.466136 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.466141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.466233 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.466343 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:23:30.467204 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.468269 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.470372 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.471944 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.472211 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.472712 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.472765 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.472869 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:23:30.473146 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.465134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36135 (local address 127.19.246.193:48099)
0810 03:23:30.465797 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:30.465812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:23:30.465980 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:23:30.466434 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:23:30.466439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:23:30.466471 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:23:30.466946 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.466964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.468437 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.468441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.470158 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:23:30.470174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.472213 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.472222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:23:30.472481 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.472519 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:23:30.472621 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 03:23:30.472774 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"spinlock_wait_cycles":20224,"thread_start_us":351,"threads_started":1}
I20260810 03:23:30.473738 5983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e664f8a27349fa8f7831538366ed3e" candidate_uuid: "18edfbfb048e4b94892801df59ea2da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" is_pre_election: true
I20260810 03:23:30.474323 6130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e664f8a27349fa8f7831538366ed3e" candidate_uuid: "18edfbfb048e4b94892801df59ea2da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d31b64fec6499fb53242bed5364223" is_pre_election: true
I20260810 03:23:30.474421 5983 raft_consensus.cc:2454] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18edfbfb048e4b94892801df59ea2da7 in term 0.
I20260810 03:23:30.474787 6130 raft_consensus.cc:2454] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18edfbfb048e4b94892801df59ea2da7 in term 0.
I20260810 03:23:30.475327 5773 leader_election.cc:302] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18edfbfb048e4b94892801df59ea2da7, c15be7827cc2401d8b6dfbfa7021c3a7; no voters:
I20260810 03:23:30.475879 6205 raft_consensus.cc:2781] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:23:30.476040 6205 raft_consensus.cc:482] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:23:30.476163 6205 raft_consensus.cc:3037] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:23:30.479050 6205 raft_consensus.cc:504] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.479811 6205 leader_election.cc:288] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [CANDIDATE]: Term 1 election: Requested vote from peers c15be7827cc2401d8b6dfbfa7021c3a7 (127.19.246.194:36433), 45d31b64fec6499fb53242bed5364223 (127.19.246.195:36135)
I20260810 03:23:30.480451 5983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e664f8a27349fa8f7831538366ed3e" candidate_uuid: "18edfbfb048e4b94892801df59ea2da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7"
I20260810 03:23:30.480549 6130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e664f8a27349fa8f7831538366ed3e" candidate_uuid: "18edfbfb048e4b94892801df59ea2da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d31b64fec6499fb53242bed5364223"
I20260810 03:23:30.480702 5983 raft_consensus.cc:3037] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:23:30.480878 6130 raft_consensus.cc:3037] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:23:30.483119 5983 raft_consensus.cc:2454] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18edfbfb048e4b94892801df59ea2da7 in term 1.
I20260810 03:23:30.483619 6130 raft_consensus.cc:2454] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18edfbfb048e4b94892801df59ea2da7 in term 1.
I20260810 03:23:30.483704 5773 leader_election.cc:302] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18edfbfb048e4b94892801df59ea2da7, c15be7827cc2401d8b6dfbfa7021c3a7; no voters:
I20260810 03:23:30.484170 6205 raft_consensus.cc:2781] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:23:30.484771 6205 raft_consensus.cc:686] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 1 LEADER]: Becoming Leader. State: Replica: 18edfbfb048e4b94892801df59ea2da7, State: Running, Role: LEADER
I20260810 03:23:30.485401 6205 consensus_queue.cc:237] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } }
I20260810 03:23:30.490743 5693 catalog_manager.cc:5495] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18edfbfb048e4b94892801df59ea2da7 (127.19.246.193). New cstate: current_term: 1 leader_uuid: "18edfbfb048e4b94892801df59ea2da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18edfbfb048e4b94892801df59ea2da7" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 33771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 } health_report { overall_health: UNKNOWN } } }
I20260810 03:23:30.563639 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.557200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.19.246.193:33771)
0810 03:23:30.557417 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.557422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.557439 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:23:30.557535 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.557539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.557628 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.557765 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:23:30.558259 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.559100 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.561318 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.561766 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.561866 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.562398 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:23:30.562957 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:23:30.563072 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.563184 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.563283 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:23:30.573753 6130 raft_consensus.cc:1261] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 1 FOLLOWER]: Refusing update from remote peer 18edfbfb048e4b94892801df59ea2da7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:23:30.573839 5983 raft_consensus.cc:1261] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 1 FOLLOWER]: Refusing update from remote peer 18edfbfb048e4b94892801df59ea2da7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:23:30.575057 6205 consensus_queue.cc:1035] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d31b64fec6499fb53242bed5364223" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:23:30.575727 6211 consensus_queue.cc:1035] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c15be7827cc2401d8b6dfbfa7021c3a7" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 36433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:23:30.591346 6213 mvcc.cc:205] Tried to move back new op lower bound from 7316816734497636352 to 7316816734152871936. Current Snapshot: MvccSnapshot[applied={T|T < 7316816734497636352}]
I20260810 03:23:30.592881 6216 mvcc.cc:205] Tried to move back new op lower bound from 7316816734497636352 to 7316816734152871936. Current Snapshot: MvccSnapshot[applied={T|T < 7316816734497636352}]
I20260810 03:23:30.610141 5693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:23:30.610523 5693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:23:30.614526 5693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Sending DeleteTablet for 3 replicas of tablet 13e664f8a27349fa8f7831538366ed3e
I20260810 03:23:30.616115 5821 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e664f8a27349fa8f7831538366ed3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:23:30 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:38136
I20260810 03:23:30.616258 5962 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e664f8a27349fa8f7831538366ed3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:23:30 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:48636
I20260810 03:23:30.616639 6107 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e664f8a27349fa8f7831538366ed3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:23:30 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:60986
I20260810 03:23:30.617914 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 5734
I20260810 03:23:30.633414 6225 tablet_replica.cc:330] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: stopping tablet replica
I20260810 03:23:30.633414 6228 tablet_replica.cc:330] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: stopping tablet replica
I20260810 03:23:30.634002 6228 raft_consensus.cc:2229] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:23:30.634100 6225 raft_consensus.cc:2229] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:23:30.634644 6228 raft_consensus.cc:2258] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:23:30.635710 6225 raft_consensus.cc:2258] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:23:30.636819 6227 tablet_replica.cc:330] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: stopping tablet replica
I20260810 03:23:30.637421 6227 raft_consensus.cc:2229] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:23:30.638170 6228 ts_tablet_manager.cc:1905] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:23:30.638316 6227 raft_consensus.cc:2258] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:23:30.638633 6225 ts_tablet_manager.cc:1905] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:23:30.640623 6227 ts_tablet_manager.cc:1905] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:23:30.645763 6225 ts_tablet_manager.cc:1918] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:23:30.645975 6225 log.cc:1196] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/13e664f8a27349fa8f7831538366ed3e
I20260810 03:23:30.646071 6228 ts_tablet_manager.cc:1918] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:23:30.646308 6228 log.cc:1196] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/13e664f8a27349fa8f7831538366ed3e
I20260810 03:23:30.646606 6225 ts_tablet_manager.cc:1939] T 13e664f8a27349fa8f7831538366ed3e P c15be7827cc2401d8b6dfbfa7021c3a7: Deleting consensus metadata
I20260810 03:23:30.646862 6228 ts_tablet_manager.cc:1939] T 13e664f8a27349fa8f7831538366ed3e P 45d31b64fec6499fb53242bed5364223: Deleting consensus metadata
I20260810 03:23:30.648952 5678 catalog_manager.cc:4841] TS 45d31b64fec6499fb53242bed5364223 (127.19.246.195:36135): tablet 13e664f8a27349fa8f7831538366ed3e (table test-table [id=7584b22d7c9c4d0a9c8620c36e6214e8]) successfully deleted
I20260810 03:23:30.649310 6227 ts_tablet_manager.cc:1918] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:23:30.649559 6227 log.cc:1196] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/13e664f8a27349fa8f7831538366ed3e
I20260810 03:23:30.650007 5678 catalog_manager.cc:4841] TS c15be7827cc2401d8b6dfbfa7021c3a7 (127.19.246.194:36433): tablet 13e664f8a27349fa8f7831538366ed3e (table test-table [id=7584b22d7c9c4d0a9c8620c36e6214e8]) successfully deleted
I20260810 03:23:30.650297 6227 ts_tablet_manager.cc:1939] T 13e664f8a27349fa8f7831538366ed3e P 18edfbfb048e4b94892801df59ea2da7: Deleting consensus metadata
I20260810 03:23:30.652077 5678 catalog_manager.cc:4841] TS 18edfbfb048e4b94892801df59ea2da7 (127.19.246.193:33771): tablet 13e664f8a27349fa8f7831538366ed3e (table test-table [id=7584b22d7c9c4d0a9c8620c36e6214e8]) successfully deleted
I20260810 03:23:30.653808 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.619535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.19.246.193:33771)
0810 03:23:30.619782 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.619792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.626642 (+ 6850us) server_negotiation.cc:407] Connection header received
0810 03:23:30.627930 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.627941 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.628085 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.628187 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:30.634491 (+ 6304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.635926 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.638482 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.639267 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.639417 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.643978 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:30.643996 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:30.644017 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:30.644073 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:30.649809 (+ 5736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.651328 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.651340 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.651343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.651598 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.652116 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.652122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.652124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.652579 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:30.652900 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.653237 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.653464 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:23:30.654558 5885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:30.760620 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 5734
I20260810 03:23:30.782600 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 5894
I20260810 03:23:30.798555 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.783485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48648 (local address 127.19.246.194:36433)
0810 03:23:30.783721 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.783787 (+ 66us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.783812 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:23:30.784075 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.784081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.784162 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.784297 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:30.785723 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.786483 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.788048 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.788498 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.788652 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.791717 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:30.791740 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:30.791757 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:30.791803 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:30.795026 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.796052 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.796062 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.796064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.796268 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.796753 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.796757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.796759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.797182 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:30.797561 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.797914 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.798173 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:23:30.799494 6027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:30.897816 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 5894
I20260810 03:23:30.920439 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 6037
I20260810 03:23:30.936748 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:30.921341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.19.246.195:36135)
0810 03:23:30.921517 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:23:30.921522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:23:30.921542 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:23:30.921700 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:23:30.921705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:23:30.921772 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:23:30.921861 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:23:30.923912 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.924767 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.926296 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:23:30.926792 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:23:30.926920 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:23:30.930210 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:23:30.930236 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:23:30.930253 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:23:30.930322 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:23:30.933500 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.934390 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.934398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.934400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.934556 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:23:30.934998 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:23:30.935001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:23:30.935003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:23:30.935541 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:23:30.935788 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 03:23:30.936116 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 03:23:30.936344 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:23:30.937695 6173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:31.031971 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 6037
I20260810 03:23:31.052784 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 5657
I20260810 03:23:31.054076 5727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:23:31.173391 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 5657
I20260810 03:23:51.199337 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36501
--webserver_interface=127.19.246.254
--webserver_port=40991
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:23:51.647294 6235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:51.647804 6235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:51.647974 6235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:51.655748 6235 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:23:51.655907 6235 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:23:51.655983 6235 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:23:51.656087 6235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:23:51.656196 6235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:51.656304 6235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:51.656396 6235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:23:51.656479 6235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:23:51.667326 6235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:51.668857 6235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:51.670378 6235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:51.684689 6242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:51.684656 6245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:51.684579 6243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:51.685073 6235 server_base.cc:1034] running on GCE node
I20260810 03:23:51.686353 6235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:51.688081 6235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:51.689424 6235 hybrid_clock.cc:648] HybridClock initialized: now 1786332231689312 us; error 61 us; skew 500 ppm
I20260810 03:23:51.689873 6235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:51 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332231, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:51.698630 6235 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:23:51.700448 6235 webserver.cc:460] Webserver started at http://127.19.246.254:40991/ using document root <none> and password file <none>
I20260810 03:23:51.701301 6235 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:51.701473 6235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:51.707724 6235 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:23:51.711197 6252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:51.713110 6235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:23:51.713366 6235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "52e84229034a4933bc0c2854b206f8c8"
format_stamp: "Formatted at 2026-08-10 03:23:04 on dist-test-slave-btw8"
server_key: "7d70dd2e6f7f3d08880bc5450234b576"
server_key_iv: "3b1dd65161dfedcd49955e40539c46a6"
server_key_version: "encryptionkey@0"
I20260810 03:23:51.714475 6235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:51.733130 6235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:51.738660 6235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:51.739025 6235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:51.761771 6235 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:36501
I20260810 03:23:51.761811 6308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:36501 every 8 connection(s)
I20260810 03:23:51.763571 6235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:23:51.771132 6309 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:23:51.771847 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 6235
I20260810 03:23:51.772970 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:33771
--local_ip_for_outbound_sockets=127.19.246.193
--tserver_master_addrs=127.19.246.254:36501
--webserver_port=40655
--webserver_interface=127.19.246.193
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:23:51.776598 6309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Bootstrap starting.
I20260810 03:23:51.803099 6309 log.cc:824] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:23:51.825027 6309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:23:51.826026 6309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Bootstrap complete.
I20260810 03:23:51.837383 6309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:51.838471 6309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e84229034a4933bc0c2854b206f8c8, State: Initialized, Role: FOLLOWER
I20260810 03:23:51.839249 6309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:51.839622 6309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:23:51.839799 6309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:23:51.840032 6309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:23:51.845577 6309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:51.846654 6309 leader_election.cc:302] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e84229034a4933bc0c2854b206f8c8; no voters:
I20260810 03:23:51.847683 6309 leader_election.cc:288] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 03:23:51.847854 6313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:23:51.849575 6313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [term 3 LEADER]: Becoming Leader. State: Replica: 52e84229034a4933bc0c2854b206f8c8, State: Running, Role: LEADER
I20260810 03:23:51.850504 6309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:23:51.850356 6313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } }
I20260810 03:23:51.853700 6314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "52e84229034a4933bc0c2854b206f8c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } } }
I20260810 03:23:51.854259 6314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:23:51.853700 6315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e84229034a4933bc0c2854b206f8c8. Latest consensus state: current_term: 3 leader_uuid: "52e84229034a4933bc0c2854b206f8c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e84229034a4933bc0c2854b206f8c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 36501 } } }
I20260810 03:23:51.854959 6315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:23:51.865453 6319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:23:51.877480 6319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:23:51.879475 6319 catalog_manager.cc:1259] Loaded cluster ID: 61871238f04346a298cecd6283bb0c12
I20260810 03:23:51.880199 6319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:23:51.884999 6319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:23:51.887902 6319 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Loaded TSK: 0
I20260810 03:23:51.888229 6319 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52e84229034a4933bc0c2854b206f8c8: Loaded TSK: 1
F20260810 03:23:51.889556 6319 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f63fa0139ad google::LogMessage::Fail() at ??:0
@ 0x7f63fa01688a google::LogMessage::SendToLog() at ??:0
@ 0x7f63fa0134de google::LogMessage::Flush() at ??:0
@ 0x7f63fa014a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f63fdd777d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f63fdd6673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f640fa421c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f640fa41783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f640facd2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f640facd0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f640fb175e9 std::function<>::operator()() at ??:0
@ 0x7f640fa45cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f640fa43c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f640facb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f640facb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f63fcc5f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f63fcc6291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f63fcc62799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f63fcc3deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f63fff176db start_thread at ??:0
@ 0x7f63f8ead61f clone at ??:0
W20260810 03:23:52.256595 6311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:52.256932 6311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:52.257117 6311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:52.265491 6311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:52.265653 6311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:52.265864 6311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:23:52.277428 6311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:33771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=40655
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:52.278934 6311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:52.280400 6311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:52.295506 6358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:52.295542 6359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:52.295594 6361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:52.296276 6311 server_base.cc:1034] running on GCE node
I20260810 03:23:52.297021 6311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:52.298148 6311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:52.299458 6311 hybrid_clock.cc:648] HybridClock initialized: now 1786332232299386 us; error 61 us; skew 500 ppm
I20260810 03:23:52.299827 6311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:52 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332232, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:52.303531 6311 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:23:52.305320 6311 webserver.cc:460] Webserver started at http://127.19.246.193:40655/ using document root <none> and password file <none>
I20260810 03:23:52.306120 6311 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:52.306314 6311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:52.313261 6311 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:23:52.316571 6369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:52.318220 6311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:23:52.318427 6311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "18edfbfb048e4b94892801df59ea2da7"
format_stamp: "Formatted at 2026-08-10 03:23:05 on dist-test-slave-btw8"
server_key: "5700d0f985c774fe9612a0962a5e45bd"
server_key_iv: "d4fc97956d41d2870ccfeb8ddaceca0f"
server_key_version: "encryptionkey@0"
I20260810 03:23:52.319604 6311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:52.345359 6311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:52.350188 6311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:52.350502 6311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:52.351760 6311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:52.353729 6311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:52.353845 6311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:52.354031 6311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:52.354133 6311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:52.392652 6311 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:33771
I20260810 03:23:52.392786 6489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:33771 every 8 connection(s)
I20260810 03:23:52.394323 6311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:23:52.397729 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 6311
I20260810 03:23:52.398552 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:36433
--local_ip_for_outbound_sockets=127.19.246.194
--tserver_master_addrs=127.19.246.254:36501
--webserver_port=45451
--webserver_interface=127.19.246.194
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:23:52.399814 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:52.397558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:42723)
0810 03:23:52.398448 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:52.398773 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":361,"threads_started":1}
W20260810 03:23:52.400627 6490 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:36501 (0 consecutive failures): Network error: Failed to ping master at 127.19.246.254:36501: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
I20260810 03:23:52.401785 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:52.401361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:43053)
0810 03:23:52.401536 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:52.401616 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 03:23:52.402704 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:52.402342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:38235)
0810 03:23:52.402491 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:52.402580 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 03:23:52.402936 6490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:23:52.766196 6494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:52.766542 6494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:52.766644 6494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:52.774662 6494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:52.774828 6494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:52.775028 6494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:23:52.786088 6494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:36433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=45451
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:52.787626 6494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:52.788986 6494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:52.803983 6499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:52.804042 6500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:52.804114 6502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:52.805573 6494 server_base.cc:1034] running on GCE node
I20260810 03:23:52.806279 6494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:52.807449 6494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:52.808758 6494 hybrid_clock.cc:648] HybridClock initialized: now 1786332232808699 us; error 57 us; skew 500 ppm
I20260810 03:23:52.809073 6494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:52 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332232, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:52.812923 6494 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:23:52.814659 6494 webserver.cc:460] Webserver started at http://127.19.246.194:45451/ using document root <none> and password file <none>
I20260810 03:23:52.815523 6494 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:52.815742 6494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:52.821722 6494 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:23:52.825167 6509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:52.826994 6494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:23:52.827462 6494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "c15be7827cc2401d8b6dfbfa7021c3a7"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "f987734935419023d215659c57f4213d"
server_key_iv: "88554428522df38826e631e158102c44"
server_key_version: "encryptionkey@0"
I20260810 03:23:52.828624 6494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:52.860594 6494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:52.865747 6494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:52.866015 6494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:52.867030 6494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:52.868633 6494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:52.868734 6494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:52.868821 6494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:52.868881 6494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:52.905515 6494 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:36433
I20260810 03:23:52.905601 6638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:36433 every 8 connection(s)
I20260810 03:23:52.907064 6494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:23:52.912627 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:52.910490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:38315)
0810 03:23:52.911416 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:52.911706 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":332,"threads_started":1}
W20260810 03:23:52.913548 6639 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:36501 (0 consecutive failures): Network error: Failed to ping master at 127.19.246.254:36501: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
I20260810 03:23:52.914546 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:52.914132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:37643)
0810 03:23:52.914315 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:52.914396 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 03:23:52.915244 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 6494
I20260810 03:23:52.915702 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:52.915242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:44849)
0810 03:23:52.915450 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:52.915548 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 03:23:52.915995 6639 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:23:52.916267 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:36135
--local_ip_for_outbound_sockets=127.19.246.195
--tserver_master_addrs=127.19.246.254:36501
--webserver_port=42391
--webserver_interface=127.19.246.195
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:23:53.290491 6642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:23:53.290884 6642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:23:53.291062 6642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:23:53.300380 6642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:23:53.300534 6642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:23:53.300674 6642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:23:53.312341 6642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:36135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=42391
--tserver_master_addrs=127.19.246.254:36501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:23:53.313759 6642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:23:53.315088 6642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:23:53.329897 6647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:53.329923 6648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:23:53.330184 6650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:23:53.331162 6642 server_base.cc:1034] running on GCE node
I20260810 03:23:53.331977 6642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:23:53.333191 6642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:23:53.334450 6642 hybrid_clock.cc:648] HybridClock initialized: now 1786332233334346 us; error 94 us; skew 500 ppm
I20260810 03:23:53.334769 6642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:23:53 dist-test-slave-btw8 krb5kdc[5051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332233, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:23:53.338428 6642 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:23:53.340340 6642 webserver.cc:460] Webserver started at http://127.19.246.195:42391/ using document root <none> and password file <none>
I20260810 03:23:53.341102 6642 fs_manager.cc:359] Metadata directory not provided
I20260810 03:23:53.341228 6642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:23:53.347589 6642 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:23:53.350924 6657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:53.352778 6642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:23:53.353024 6642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "45d31b64fec6499fb53242bed5364223"
format_stamp: "Formatted at 2026-08-10 03:23:06 on dist-test-slave-btw8"
server_key: "054ff97e83387faa38d14f62bb9fedf3"
server_key_iv: "ca7590cbb44c1a326ffab8ceafd1d8d0"
server_key_version: "encryptionkey@0"
I20260810 03:23:53.354277 6642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:23:53.374990 6642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:23:53.380439 6642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:23:53.380841 6642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:23:53.382236 6642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:23:53.384212 6642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:23:53.384362 6642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:53.384488 6642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:23:53.384590 6642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:23:53.404768 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:53.403727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:50903)
0810 03:23:53.404401 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:53.404505 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":383,"threads_started":1}
I20260810 03:23:53.425289 6642 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:36135
I20260810 03:23:53.425393 6782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:36135 every 8 connection(s)
I20260810 03:23:53.426940 6642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:23:53.432976 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:53.430511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:56751)
0810 03:23:53.431520 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:53.431807 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":360,"threads_started":1}
I20260810 03:23:53.433291 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 6642
W20260810 03:23:53.433805 6784 heartbeater.cc:644] Failed to heartbeat to 127.19.246.254:36501 (0 consecutive failures): Network error: Failed to ping master at 127.19.246.254:36501: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
I20260810 03:23:53.435071 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:53.434519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:39621)
0810 03:23:53.434772 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:53.434904 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 03:23:53.436376 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:53.435933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:58561)
0810 03:23:53.436133 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:53.436242 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260810 03:23:53.436729 6784 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:23:53.917938 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:53.916944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:60069)
0810 03:23:53.917553 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:53.917676 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":322,"threads_started":1}
I20260810 03:23:54.406901 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:54.406026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:47803)
0810 03:23:54.406603 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:54.406714 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":307,"threads_started":1}
I20260810 03:23:54.438596 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:54.437627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:50715)
0810 03:23:54.438199 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:54.438321 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":293,"threads_started":1}
I20260810 03:23:54.920254 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:54.919316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:47377)
0810 03:23:54.919962 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:54.920061 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":338,"threads_started":1}
I20260810 03:23:55.409200 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:55.408086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:47665)
0810 03:23:55.408833 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:55.408940 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":349,"threads_started":1}
I20260810 03:23:55.440788 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:55.439867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:45703)
0810 03:23:55.440479 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:55.440594 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":331,"threads_started":1}
I20260810 03:23:55.922201 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:55.921315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:36811)
0810 03:23:55.921928 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:55.922017 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":392,"threads_started":1}
I20260810 03:23:56.411505 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:56.410449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:33255)
0810 03:23:56.411081 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:56.411183 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":316,"threads_started":1}
I20260810 03:23:56.442888 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:56.442107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:60231)
0810 03:23:56.442605 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:56.442705 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260810 03:23:56.924387 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:56.923342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:54177)
0810 03:23:56.924066 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:56.924179 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":298,"threads_started":1}
I20260810 03:23:57.413851 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:57.412958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:46785)
0810 03:23:57.413528 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:57.413637 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":310,"threads_started":1}
I20260810 03:23:57.444916 6801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:57.444085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:43995)
0810 03:23:57.444631 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:57.444735 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":287,"threads_started":1}
I20260810 03:23:57.926537 6802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:57.925642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:55707)
0810 03:23:57.926215 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:57.926334 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":326,"threads_started":1}
I20260810 03:23:58.416041 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:58.415102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:34427)
0810 03:23:58.415743 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:58.415863 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":367,"threads_started":1}
I20260810 03:23:58.447041 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:58.446058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:58435)
0810 03:23:58.446704 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:58.446801 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":369,"threads_started":1}
I20260810 03:23:58.928516 6805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:58.927652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:49569)
0810 03:23:58.928249 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:58.928348 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":310,"threads_started":1}
I20260810 03:23:59.418202 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:59.417241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:57229)
0810 03:23:59.417818 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:59.417919 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":258,"threads_started":1}
I20260810 03:23:59.449083 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:59.448271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:45291)
0810 03:23:59.448791 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:59.448893 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":267,"threads_started":1}
I20260810 03:23:59.930763 6809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:23:59.929789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:50077)
0810 03:23:59.930390 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 03:23:59.930500 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":270,"threads_started":1}
I20260810 03:24:00.420481 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:00.419474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:40125)
0810 03:24:00.420165 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:00.420287 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":320,"threads_started":1}
I20260810 03:24:00.451545 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:00.450290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:56117)
0810 03:24:00.451034 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:00.451155 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":381,"threads_started":1}
I20260810 03:24:00.933106 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:00.932152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:36941)
0810 03:24:00.932809 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:00.932916 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":384,"threads_started":1}
I20260810 03:24:01.422642 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:01.421767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:58605)
0810 03:24:01.422330 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:01.422428 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":257,"threads_started":1}
I20260810 03:24:01.453955 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:01.452876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:42883)
0810 03:24:01.453643 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:01.453755 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":478,"threads_started":1}
I20260810 03:24:01.935259 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:01.934295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:56781)
0810 03:24:01.934939 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:01.935065 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":345,"threads_started":1}
I20260810 03:24:02.424887 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:02.423937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:40917)
0810 03:24:02.424605 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:02.424709 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":441,"threads_started":1}
I20260810 03:24:02.456084 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:02.455118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:56149)
0810 03:24:02.455793 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:02.455902 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":376,"threads_started":1}
I20260810 03:24:02.937391 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:02.936526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:48205)
0810 03:24:02.937101 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:02.937203 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":347,"threads_started":1}
I20260810 03:24:03.426952 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:03.426054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:41235)
0810 03:24:03.426601 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:03.426745 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":280,"threads_started":1}
I20260810 03:24:03.458238 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:03.457414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:59217)
0810 03:24:03.457959 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:03.458057 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":307,"threads_started":1}
I20260810 03:24:03.939473 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:03.938502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:42911)
0810 03:24:03.939092 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:03.939198 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":340,"threads_started":1}
I20260810 03:24:04.429028 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:04.428021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:48587)
0810 03:24:04.428769 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:04.428859 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":284,"threads_started":1}
I20260810 03:24:04.460572 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:04.459671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:40881)
0810 03:24:04.460198 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:04.460316 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":257,"threads_started":1}
I20260810 03:24:04.941370 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:04.940614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:49951)
0810 03:24:04.941105 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:04.941203 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260810 03:24:05.430948 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:05.430075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:37183)
0810 03:24:05.430633 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:05.430741 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":252,"threads_started":1}
I20260810 03:24:05.462541 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:05.461674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:34055)
0810 03:24:05.462242 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:05.462363 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":289,"threads_started":1}
I20260810 03:24:05.943517 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:05.942547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:46465)
0810 03:24:05.943127 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:05.943244 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":284,"threads_started":1}
I20260810 03:24:06.433319 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:06.432468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:36291)
0810 03:24:06.432986 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:06.433076 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":248,"threads_started":1}
I20260810 03:24:06.464877 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:06.463768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:34485)
0810 03:24:06.464453 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:06.464592 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":360,"threads_started":1}
I20260810 03:24:06.945394 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:06.944607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:53875)
0810 03:24:06.945097 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:06.945199 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260810 03:24:07.435933 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:07.434664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:58005)
0810 03:24:07.435483 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:07.435718 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":444,"threads_started":1}
I20260810 03:24:07.466954 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:07.466081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:41409)
0810 03:24:07.466642 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:07.466742 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":281,"threads_started":1}
I20260810 03:24:07.947696 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:07.946790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.194:49819)
0810 03:24:07.947330 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:07.947501 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":299,"threads_started":1}
I20260810 03:24:08.438161 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:08.437072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.193:44645)
0810 03:24:08.437823 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:08.437962 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":323,"threads_started":1}
I20260810 03:24:08.441553 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 6311
I20260810 03:24:08.469291 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:08.443265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.19.246.193:33771)
0810 03:24:08.444142 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:08.444160 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:08.444209 (+ 49us) server_negotiation.cc:407] Connection header received
0810 03:24:08.445956 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:08.445968 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:08.446434 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:08.446789 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:08.448672 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:08.450063 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:08.451604 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:08.452394 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:08.452549 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:08.460812 (+ 8263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:08.460824 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:08.460844 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:08.460904 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:08.464661 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:08.465630 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:08.465639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:08.465646 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:08.465860 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:08.466534 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:08.466538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:08.466540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:08.467299 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:08.467785 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:08.468293 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 03:24:08.468955 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":482,"threads_started":1}
I20260810 03:24:08.469324 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:08.468389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:36501 (local address 127.19.246.195:36955)
0810 03:24:08.469001 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:08.469116 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.254:36501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":328,"threads_started":1}
I20260810 03:24:08.470948 6485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:08.562498 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 6311
I20260810 03:24:08.585676 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 6494
I20260810 03:24:08.605841 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:08.586735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.19.246.194:36433)
0810 03:24:08.587318 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:08.587335 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:08.587443 (+ 108us) server_negotiation.cc:407] Connection header received
0810 03:24:08.589152 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:08.589163 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:08.589603 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:08.590003 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:08.591857 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:08.593153 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:08.594460 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:08.595187 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:08.595441 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:08.598511 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:08.598520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:08.598539 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:08.598599 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:08.601887 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:08.602785 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:08.602793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:08.602800 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:08.603002 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:08.603522 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:08.603527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:08.603536 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:08.604135 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:08.604509 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:08.605036 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 03:24:08.605519 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":270,"threads_started":1}
I20260810 03:24:08.607679 6634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:08.695046 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 6494
I20260810 03:24:08.716722 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 6642
I20260810 03:24:08.742026 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:08.718092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.19.246.195:36135)
0810 03:24:08.718873 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:08.718888 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:08.718933 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:24:08.720763 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:08.720775 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:08.721275 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:08.721664 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:08.723283 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:08.724696 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:08.726107 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:08.726826 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:08.727046 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:08.734135 (+ 7089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:08.734147 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:08.734169 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:08.734235 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:08.738062 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:08.738991 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:08.738999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:08.739007 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:08.739212 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:08.739734 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:08.739740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:08.739744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:08.740373 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:08.740709 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:08.741232 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 03:24:08.741710 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":412,"threads_started":1}
I20260810 03:24:08.743734 6778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:08.858976 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 6642
2026-08-10T03:24:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66006 ms)
[----------] 30 tests from SecurityITest (343354 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:08 dist-test-slave-btw8 krb5kdc[6848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:08 dist-test-slave-btw8 krb5kdc[6848](info): set up 2 sockets
Aug 10 03:24:08 dist-test-slave-btw8 krb5kdc[6848](info): commencing operation
krb5kdc: starting...
W20260810 03:24:10.122651 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:10 dist-test-slave-btw8 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:24:10.506047 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41255
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:10.899171 6861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:10.899660 6861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:10.899837 6861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:10.907823 6861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:10.907965 6861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:10.908015 6861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:10.908068 6861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:10.908114 6861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:10.919536 6861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:10.921200 6861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:10.922729 6861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:10.938269 6870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:10.938344 6868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:10.938243 6867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:10.938999 6861 server_base.cc:1034] running on GCE node
I20260810 03:24:10.940074 6861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:10.941830 6861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:10.943115 6861 hybrid_clock.cc:648] HybridClock initialized: now 1786332250943088 us; error 117 us; skew 500 ppm
I20260810 03:24:10.943598 6861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:10 dist-test-slave-btw8 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332250, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:10.947288 6861 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:24:10.949275 6861 webserver.cc:460] Webserver started at http://127.19.246.254:40107/ using document root <none> and password file <none>
I20260810 03:24:10.950091 6861 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:10.950290 6861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:10.950642 6861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:10.954197 6861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "8162a69f6d374430b6660f30f031549f"
format_stamp: "Formatted at 2026-08-10 03:24:10 on dist-test-slave-btw8"
server_key: "d9ccaa83d9b2de00cb5040997853c6ed"
server_key_iv: "f5b58701d952a09379b45c66647eb457"
server_key_version: "encryptionkey@0"
I20260810 03:24:10.955083 6861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "8162a69f6d374430b6660f30f031549f"
format_stamp: "Formatted at 2026-08-10 03:24:10 on dist-test-slave-btw8"
server_key: "d9ccaa83d9b2de00cb5040997853c6ed"
server_key_iv: "f5b58701d952a09379b45c66647eb457"
server_key_version: "encryptionkey@0"
I20260810 03:24:10.961545 6861 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:24:10.966248 6877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:10.968116 6861 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:24:10.968346 6861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "8162a69f6d374430b6660f30f031549f"
format_stamp: "Formatted at 2026-08-10 03:24:10 on dist-test-slave-btw8"
server_key: "d9ccaa83d9b2de00cb5040997853c6ed"
server_key_iv: "f5b58701d952a09379b45c66647eb457"
server_key_version: "encryptionkey@0"
I20260810 03:24:10.968600 6861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:10.996587 6861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:11.001811 6861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:11.002202 6861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:11.024134 6861 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:41255
I20260810 03:24:11.024173 6936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:41255 every 8 connection(s)
I20260810 03:24:11.025754 6861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:11.027544 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 6861
I20260810 03:24:11.027783 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:11.028532 20443 external_mini_cluster.cc:1428] Setting key f3e680a9f398f42ae17a6ab35279ecc7
I20260810 03:24:11.030728 6937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:11.046200 6937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f: Bootstrap starting.
Aug 10 03:24:11 dist-test-slave-btw8 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:11.050437 6937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:11.051813 6937 log.cc:824] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:11.055123 6937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f: No bootstrap required, opened a new log
I20260810 03:24:11.056973 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:11.030787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.19.246.254:41255)
0810 03:24:11.031816 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:11.031833 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:11.031896 (+ 63us) server_negotiation.cc:407] Connection header received
0810 03:24:11.034367 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:11.034403 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:11.035008 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:11.035564 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:11.037294 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:11.039629 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:11.041414 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:11.042658 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:11.042865 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:11.048299 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:11.048321 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:11.048348 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:11.048403 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:11.052346 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:11.053240 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:11.053248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:11.053255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:11.053452 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:11.053905 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:11.053912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:11.053919 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:11.054508 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:11.054853 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:11.055423 (+ 570us) server_negotiation.cc:299] Negotiation successful
0810 03:24:11.055928 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":540,"threads_started":1}
I20260810 03:24:11.064723 6937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8162a69f6d374430b6660f30f031549f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41255 } }
I20260810 03:24:11.065310 6937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:11.065444 6937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8162a69f6d374430b6660f30f031549f, State: Initialized, Role: FOLLOWER
I20260810 03:24:11.066272 6937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8162a69f6d374430b6660f30f031549f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41255 } }
I20260810 03:24:11.066625 6937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:11.066787 6937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:11.066931 6937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:11.069406 6937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8162a69f6d374430b6660f30f031549f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41255 } }
I20260810 03:24:11.069973 6937 leader_election.cc:302] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8162a69f6d374430b6660f30f031549f; no voters:
I20260810 03:24:11.070771 6937 leader_election.cc:288] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:11.071000 6943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:11.071882 6943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [term 1 LEADER]: Becoming Leader. State: Replica: 8162a69f6d374430b6660f30f031549f, State: Running, Role: LEADER
I20260810 03:24:11.072964 6943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8162a69f6d374430b6660f30f031549f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41255 } }
I20260810 03:24:11.073632 6937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:11.077337 6945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8162a69f6d374430b6660f30f031549f. Latest consensus state: current_term: 1 leader_uuid: "8162a69f6d374430b6660f30f031549f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8162a69f6d374430b6660f30f031549f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41255 } } }
I20260810 03:24:11.077940 6945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:11.078639 6944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8162a69f6d374430b6660f30f031549f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8162a69f6d374430b6660f30f031549f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41255 } } }
I20260810 03:24:11.078982 6944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:11.079077 6953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:11.086431 6953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:11.099784 6953 catalog_manager.cc:1347] Generated new cluster ID: ffd494418ebd4eb6af10def586e70bf9
I20260810 03:24:11.099920 6953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:11.112746 6953 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:11.114300 6953 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:11.134775 6953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f: Generated new TSK 0
I20260810 03:24:11.136113 6953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:24:11.232609 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41255
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:11.639026 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:11.639430 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:11.639612 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:11.647321 6966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:11.647598 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:11.647799 6966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:24:11.659581 6966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:11.661161 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:11.662640 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:11.678826 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:11.678998 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:11.679457 6966 server_base.cc:1034] running on GCE node
W20260810 03:24:11.679517 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:11.680512 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:11.681670 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:11.682963 6966 hybrid_clock.cc:648] HybridClock initialized: now 1786332251682916 us; error 42 us; skew 500 ppm
I20260810 03:24:11.683422 6966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:11 dist-test-slave-btw8 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332251, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:11.687033 6966 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:24:11.688967 6966 webserver.cc:460] Webserver started at http://127.19.246.193:36391/ using document root <none> and password file <none>
I20260810 03:24:11.689746 6966 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:11.689914 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:11.690287 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:11.693583 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "d79e0bc2222843bea57748285363e322"
format_stamp: "Formatted at 2026-08-10 03:24:11 on dist-test-slave-btw8"
server_key: "0a4d05dfd392c81d30d3434f0d58ff7c"
server_key_iv: "bfb2cd220c2b05fcb10e5e41414e7e83"
server_key_version: "encryptionkey@0"
I20260810 03:24:11.694439 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "d79e0bc2222843bea57748285363e322"
format_stamp: "Formatted at 2026-08-10 03:24:11 on dist-test-slave-btw8"
server_key: "0a4d05dfd392c81d30d3434f0d58ff7c"
server_key_iv: "bfb2cd220c2b05fcb10e5e41414e7e83"
server_key_version: "encryptionkey@0"
I20260810 03:24:11.700866 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260810 03:24:11.705466 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:11.707238 6966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 03:24:11.707618 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "d79e0bc2222843bea57748285363e322"
format_stamp: "Formatted at 2026-08-10 03:24:11 on dist-test-slave-btw8"
server_key: "0a4d05dfd392c81d30d3434f0d58ff7c"
server_key_iv: "bfb2cd220c2b05fcb10e5e41414e7e83"
server_key_version: "encryptionkey@0"
I20260810 03:24:11.707932 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:11.727869 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:11.732997 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:11.733337 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:11.734436 6966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:11.736104 6966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:11.736186 6966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:11.736320 6966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:11.736387 6966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:11.775596 6966 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:40799
I20260810 03:24:11.775605 7102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:40799 every 8 connection(s)
I20260810 03:24:11.777186 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:24:11.784022 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 6966
I20260810 03:24:11.784325 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:24:11.785089 20443 external_mini_cluster.cc:1428] Setting key 20672ff5f9b8e2371af969652772d556
Aug 10 03:24:11 dist-test-slave-btw8 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332251, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:11.805713 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:11.780774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:44593 (local address 127.19.246.254:41255)
0810 03:24:11.781457 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:11.781463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:11.782824 (+ 1361us) server_negotiation.cc:407] Connection header received
0810 03:24:11.785154 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:11.785160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:11.785275 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:11.785418 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:11.787662 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:11.788771 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:11.790750 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:11.791547 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:11.791705 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:11.796405 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:11.796415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:11.796423 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:11.796494 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:11.800755 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:11.802002 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:11.802011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:11.802017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:11.802197 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:11.802791 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:11.802800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:11.802806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:11.803248 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:11.803584 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:11.804896 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0810 03:24:11.805302 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":361,"threads_started":1}
I20260810 03:24:11.807312 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:11.781544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41255 (local address 127.19.246.193:44593)
0810 03:24:11.782602 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:11.782651 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:11.784808 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:11.785784 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:11.785798 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:11.786513 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:11.787440 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:11.787471 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:11.788974 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:11.788979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:11.790548 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:11.790563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:11.791887 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:11.791902 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:11.792151 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:11.793044 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:11.793077 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:11.795968 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:11.801008 (+ 5040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:11.801018 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:11.801036 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:11.801807 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:11.802383 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:11.802387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:11.802390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:11.802634 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:11.803482 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:11.803496 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:11.804156 (+ 660us) client_negotiation.cc:769] Sending connection context
0810 03:24:11.804596 (+ 440us) client_negotiation.cc:240] Negotiation successful
0810 03:24:11.805494 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":384,"threads_started":1}
I20260810 03:24:11.809742 7103 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41255
I20260810 03:24:11.810138 7103 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:11.811040 7103 heartbeater.cc:511] Master 127.19.246.254:41255 requested a full tablet report, sending...
I20260810 03:24:11.814354 6896 ts_manager.cc:194] Registered new tserver with Master: d79e0bc2222843bea57748285363e322 (127.19.246.193:40799)
I20260810 03:24:11.816838 6896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:44593
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:24:11.872816 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41255
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:12.273335 7110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:12.274016 7110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:12.274269 7110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:12.284044 7110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:12.284261 7110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:12.284420 7110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:24:12.296371 7110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:12.297796 7110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:12.299242 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:12.316010 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:12.316020 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:12.316066 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:12.317296 7110 server_base.cc:1034] running on GCE node
I20260810 03:24:12.318100 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:12.319424 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:12.320785 7110 hybrid_clock.cc:648] HybridClock initialized: now 1786332252320675 us; error 69 us; skew 500 ppm
I20260810 03:24:12.321314 7110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:12 dist-test-slave-btw8 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332252, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:12.325834 7110 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:24:12.327924 7110 webserver.cc:460] Webserver started at http://127.19.246.194:45765/ using document root <none> and password file <none>
I20260810 03:24:12.328748 7110 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:12.328918 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:12.329282 7110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:12.332878 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "e0fc3332a64844f69799d09297d979b0"
format_stamp: "Formatted at 2026-08-10 03:24:12 on dist-test-slave-btw8"
server_key: "d93fce3bd0a8483e8d1df192ec2462b8"
server_key_iv: "dd9edb6de42d0307024e3dfba775ce6d"
server_key_version: "encryptionkey@0"
I20260810 03:24:12.333956 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "e0fc3332a64844f69799d09297d979b0"
format_stamp: "Formatted at 2026-08-10 03:24:12 on dist-test-slave-btw8"
server_key: "d93fce3bd0a8483e8d1df192ec2462b8"
server_key_iv: "dd9edb6de42d0307024e3dfba775ce6d"
server_key_version: "encryptionkey@0"
I20260810 03:24:12.340894 7110 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260810 03:24:12.346097 7126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:12.348100 7110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:24:12.348311 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "e0fc3332a64844f69799d09297d979b0"
format_stamp: "Formatted at 2026-08-10 03:24:12 on dist-test-slave-btw8"
server_key: "d93fce3bd0a8483e8d1df192ec2462b8"
server_key_iv: "dd9edb6de42d0307024e3dfba775ce6d"
server_key_version: "encryptionkey@0"
I20260810 03:24:12.348520 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:12.364943 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:12.370281 7110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:12.370577 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:12.371752 7110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:12.373551 7110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:12.373636 7110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:12.373795 7110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:12.373879 7110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:12.411911 7110 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:41885
I20260810 03:24:12.411975 7246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:41885 every 8 connection(s)
I20260810 03:24:12.413556 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:24:12.423151 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 7110
I20260810 03:24:12.423450 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:24:12.424036 20443 external_mini_cluster.cc:1428] Setting key f315e411fa826214a737dbb8c60e4892
Aug 10 03:24:12 dist-test-slave-btw8 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332252, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:12.439713 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:12.416538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:33823 (local address 127.19.246.254:41255)
0810 03:24:12.417057 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:12.417062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:12.418159 (+ 1097us) server_negotiation.cc:407] Connection header received
0810 03:24:12.420491 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:12.420499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:12.420631 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:12.420785 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:12.422695 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:12.423712 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:12.425872 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:12.426733 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:12.426890 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:12.431320 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:12.431330 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:12.431337 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:12.431413 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:12.435546 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:12.436565 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:12.436573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:12.436579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:12.436773 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:12.437258 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:12.437266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:12.437272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:12.437698 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:12.437863 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:12.438849 (+ 986us) server_negotiation.cc:299] Negotiation successful
0810 03:24:12.439151 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":292,"threads_started":1}
I20260810 03:24:12.440593 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:12.416958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41255 (local address 127.19.246.194:33823)
0810 03:24:12.417912 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:12.417973 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:12.420134 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:12.421076 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:12.421087 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:12.421751 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:12.422446 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:12.422479 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:12.423926 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:12.423931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:12.425690 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:12.425705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:12.427014 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:12.427023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:12.427316 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:12.428186 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:12.428212 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:12.430946 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:12.435812 (+ 4866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:12.435822 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:12.435840 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:12.436405 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:12.436910 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:12.436913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:12.436915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:12.437132 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:12.437864 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:12.437872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:12.438325 (+ 453us) client_negotiation.cc:769] Sending connection context
0810 03:24:12.438702 (+ 377us) client_negotiation.cc:240] Negotiation successful
0810 03:24:12.439179 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":403,"threads_started":1}
I20260810 03:24:12.442883 7247 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41255
I20260810 03:24:12.443324 7247 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:12.444687 7247 heartbeater.cc:511] Master 127.19.246.254:41255 requested a full tablet report, sending...
I20260810 03:24:12.447181 6896 ts_manager.cc:194] Registered new tserver with Master: e0fc3332a64844f69799d09297d979b0 (127.19.246.194:41885)
I20260810 03:24:12.448661 6896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:33823
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:24:12.511339 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41255
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:38135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:24:12.820979 7103 heartbeater.cc:503] Master 127.19.246.254:41255 was elected leader, sending a full tablet report...
W20260810 03:24:12.904814 7254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:12.905162 7254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:12.905377 7254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:12.913525 7254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:12.913667 7254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:12.913882 7254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:24:12.924808 7254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:38135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:41255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:12.926306 7254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:12.927927 7254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:12.943235 7261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:12.943569 7264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:12.943943 7262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:12.945681 7254 server_base.cc:1034] running on GCE node
I20260810 03:24:12.946516 7254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:12.947748 7254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:12.949024 7254 hybrid_clock.cc:648] HybridClock initialized: now 1786332252948955 us; error 73 us; skew 500 ppm
I20260810 03:24:12.949456 7254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:12 dist-test-slave-btw8 krb5kdc[6848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332252, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:12.953650 7254 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:24:12.955885 7254 webserver.cc:460] Webserver started at http://127.19.246.195:43973/ using document root <none> and password file <none>
I20260810 03:24:12.956771 7254 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:12.956935 7254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:12.957353 7254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:12.961141 7254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "825f5f053688489198b7e8b19bfe3e74"
format_stamp: "Formatted at 2026-08-10 03:24:12 on dist-test-slave-btw8"
server_key: "c77840eec46cded3daeaf2b6c4f23c86"
server_key_iv: "8c28f21c58b058772b09cae56325400c"
server_key_version: "encryptionkey@0"
I20260810 03:24:12.962198 7254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "825f5f053688489198b7e8b19bfe3e74"
format_stamp: "Formatted at 2026-08-10 03:24:12 on dist-test-slave-btw8"
server_key: "c77840eec46cded3daeaf2b6c4f23c86"
server_key_iv: "8c28f21c58b058772b09cae56325400c"
server_key_version: "encryptionkey@0"
I20260810 03:24:12.969311 7254 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260810 03:24:12.973975 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:12.976095 7254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:24:12.976347 7254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "825f5f053688489198b7e8b19bfe3e74"
format_stamp: "Formatted at 2026-08-10 03:24:12 on dist-test-slave-btw8"
server_key: "c77840eec46cded3daeaf2b6c4f23c86"
server_key_iv: "8c28f21c58b058772b09cae56325400c"
server_key_version: "encryptionkey@0"
I20260810 03:24:12.976588 7254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:13.004849 7254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:13.011124 7254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:13.011567 7254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:13.012948 7254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:13.014586 7254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:13.014725 7254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:13.014911 7254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:13.014990 7254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:13.059643 7254 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:39461
I20260810 03:24:13.059712 7392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:39461 every 8 connection(s)
I20260810 03:24:13.061170 7254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:24:13.071465 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 7254
I20260810 03:24:13.071681 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:24:13.072350 20443 external_mini_cluster.cc:1428] Setting key ed526ac4ee46f4f9f0c0d89ceed816ac
Aug 10 03:24:13 dist-test-slave-btw8 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332252, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:13.087011 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.064324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:60427 (local address 127.19.246.254:41255)
0810 03:24:13.064952 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.064964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.065928 (+ 964us) server_negotiation.cc:407] Connection header received
0810 03:24:13.068090 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.068096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.068203 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.068337 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:13.070134 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.070974 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.073219 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.073851 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.074019 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.078739 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:13.078744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:13.078748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:13.078838 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:13.082605 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.084054 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.084063 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.084073 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.084320 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.084864 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.084874 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.084876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.085204 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:13.085386 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.086238 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.086573 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":367,"threads_started":1}
I20260810 03:24:13.088277 7396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.064781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41255 (local address 127.19.246.195:60427)
0810 03:24:13.065699 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:13.065762 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:13.067741 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:13.068621 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:13.068630 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:13.069240 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:13.069898 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.069925 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.071181 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.071186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.073005 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.073024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.074173 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.074183 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.074462 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.075566 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:13.075593 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:13.078477 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:13.082822 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:13.082832 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:13.082848 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:13.083858 (+ 1010us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:13.084463 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:13.084476 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:13.084479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:13.084709 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:13.085361 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:13.085370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:13.085801 (+ 431us) client_negotiation.cc:769] Sending connection context
0810 03:24:13.086115 (+ 314us) client_negotiation.cc:240] Negotiation successful
0810 03:24:13.086772 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":418,"threads_started":1}
I20260810 03:24:13.089933 7393 heartbeater.cc:347] Connected to a master server at 127.19.246.254:41255
I20260810 03:24:13.090327 7393 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:13.091266 7393 heartbeater.cc:511] Master 127.19.246.254:41255 requested a full tablet report, sending...
I20260810 03:24:13.093214 6895 ts_manager.cc:194] Registered new tserver with Master: 825f5f053688489198b7e8b19bfe3e74 (127.19.246.195:39461)
I20260810 03:24:13.094245 6895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:60427
I20260810 03:24:13.102746 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:24:13.125000 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.108403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.19.246.254:41255)
0810 03:24:13.108683 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.108690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.109038 (+ 348us) server_negotiation.cc:407] Connection header received
0810 03:24:13.109223 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.109227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.109322 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.109412 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:13.111339 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.112493 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.114207 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.114687 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.114834 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.117768 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:13.117774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:13.117778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:13.117820 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:13.121286 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.122321 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.122326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.122329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.122493 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.123009 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.123014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.123016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.123590 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:13.123897 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.124334 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.124589 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:24:13.130774 6895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:24:13.139513 6895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:24:13.173033 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.155584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:41885 (local address 127.0.0.1:48024)
0810 03:24:13.157211 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:13.157281 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:13.157535 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:13.159378 (+ 1843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:13.159391 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:13.159452 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:13.160220 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.160252 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.163458 (+ 3206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.163469 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.167791 (+ 4322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.167824 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.171095 (+ 3271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.171107 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.171557 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.171661 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 03:24:13.171840 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 03:24:13.172247 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1220,"mutex_wait_us":44,"thread_start_us":10502,"threads_started":1}
I20260810 03:24:13.175407 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.155517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.19.246.194:41885)
0810 03:24:13.158380 (+ 2863us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.158394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.158436 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:24:13.158616 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.158628 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.158982 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.160549 (+ 1567us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:13.160878 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.163128 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.168257 (+ 5129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.170723 (+ 2466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.173929 (+ 3206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.174366 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.174511 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.174790 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":2447,"threads_started":1}
I20260810 03:24:13.178217 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.156883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.19.246.193:40799)
0810 03:24:13.166065 (+ 9182us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.166078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.166800 (+ 722us) server_negotiation.cc:407] Connection header received
0810 03:24:13.166938 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.166949 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.167306 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.167659 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:13.169055 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.171064 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.174558 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.176691 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.176960 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.177193 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.177539 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.177740 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8941,"thread_start_us":8602,"threads_started":1}
I20260810 03:24:13.178200 7409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.156002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:40799 (local address 127.0.0.1:36284)
0810 03:24:13.166402 (+ 10400us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:13.166420 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:13.166622 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:13.168086 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:13.168094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:13.168136 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:13.168763 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.168782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.171374 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.171382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.174332 (+ 2950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.174347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.177020 (+ 2673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.177028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.177323 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.177362 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:24:13.177481 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:24:13.177662 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10278,"thread_start_us":1435,"threads_started":1}
I20260810 03:24:13.180927 7177 tablet_service.cc:1467] Processing CreateTablet for tablet a7b10815d44a4d2bbc68bebfea897aed (DEFAULT_TABLE table=test-table [id=ce70a1b2755e4cc1bd1ea97be93f2704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:13.182750 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b10815d44a4d2bbc68bebfea897aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:13.183178 7033 tablet_service.cc:1467] Processing CreateTablet for tablet a7b10815d44a4d2bbc68bebfea897aed (DEFAULT_TABLE table=test-table [id=ce70a1b2755e4cc1bd1ea97be93f2704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:13.185132 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b10815d44a4d2bbc68bebfea897aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:13.185659 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.166604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:39461 (local address 127.0.0.1:39400)
0810 03:24:13.173442 (+ 6838us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:13.173472 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:13.173644 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:13.177631 (+ 3987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:13.177636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:13.177673 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:13.177895 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.177908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.180496 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.180504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.182489 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.182510 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.184775 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.184784 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.184978 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.185015 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 03:24:13.185141 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 03:24:13.185324 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6668,"thread_start_us":9333,"threads_started":1}
I20260810 03:24:13.185866 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.167130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.19.246.195:39461)
0810 03:24:13.176920 (+ 9790us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.176928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.176963 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:24:13.177065 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.177084 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.177378 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.178224 (+ 846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:13.178583 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.180288 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.182715 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.184516 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.184801 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.184998 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.185226 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.185424 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9607,"thread_start_us":9329,"threads_started":1}
I20260810 03:24:13.191028 7324 tablet_service.cc:1467] Processing CreateTablet for tablet a7b10815d44a4d2bbc68bebfea897aed (DEFAULT_TABLE table=test-table [id=ce70a1b2755e4cc1bd1ea97be93f2704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:13.193141 7324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b10815d44a4d2bbc68bebfea897aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:13.200156 7415 tablet_bootstrap.cc:492] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: Bootstrap starting.
I20260810 03:24:13.202397 7414 tablet_bootstrap.cc:492] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Bootstrap starting.
I20260810 03:24:13.204878 7415 tablet_bootstrap.cc:654] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:13.206445 7415 log.cc:824] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:13.206735 7414 tablet_bootstrap.cc:654] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:13.208163 7414 log.cc:824] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:13.210506 7415 tablet_bootstrap.cc:492] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: No bootstrap required, opened a new log
I20260810 03:24:13.210903 7415 ts_tablet_manager.cc:1397] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260810 03:24:13.211905 7418 tablet_bootstrap.cc:492] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Bootstrap starting.
I20260810 03:24:13.216028 7414 tablet_bootstrap.cc:492] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: No bootstrap required, opened a new log
I20260810 03:24:13.216382 7414 ts_tablet_manager.cc:1397] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260810 03:24:13.216425 7418 tablet_bootstrap.cc:654] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:13.217911 7418 log.cc:824] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:13.221941 7418 tablet_bootstrap.cc:492] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: No bootstrap required, opened a new log
I20260810 03:24:13.222277 7418 ts_tablet_manager.cc:1397] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 03:24:13.222715 7415 raft_consensus.cc:348] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.223277 7415 raft_consensus.cc:374] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:13.223429 7415 raft_consensus.cc:729] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fc3332a64844f69799d09297d979b0, State: Initialized, Role: FOLLOWER
I20260810 03:24:13.224187 7415 consensus_queue.cc:260] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.225543 7247 heartbeater.cc:503] Master 127.19.246.254:41255 was elected leader, sending a full tablet report...
I20260810 03:24:13.225677 7414 raft_consensus.cc:348] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.226162 7414 raft_consensus.cc:374] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:13.226264 7414 raft_consensus.cc:729] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79e0bc2222843bea57748285363e322, State: Initialized, Role: FOLLOWER
I20260810 03:24:13.226305 7415 ts_tablet_manager.cc:1428] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260810 03:24:13.226979 7414 consensus_queue.cc:260] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.229408 7414 ts_tablet_manager.cc:1428] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 03:24:13.234086 7418 raft_consensus.cc:348] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.234526 7418 raft_consensus.cc:374] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:13.234663 7418 raft_consensus.cc:729] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825f5f053688489198b7e8b19bfe3e74, State: Initialized, Role: FOLLOWER
I20260810 03:24:13.235309 7418 consensus_queue.cc:260] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.236578 7393 heartbeater.cc:503] Master 127.19.246.254:41255 was elected leader, sending a full tablet report...
I20260810 03:24:13.237277 7418 ts_tablet_manager.cc:1428] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260810 03:24:13.280506 7105 tablet.cc:2369] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:13.304589 7424 raft_consensus.cc:482] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:24:13.305050 7424 raft_consensus.cc:504] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.307039 7424 leader_election.cc:288] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0fc3332a64844f69799d09297d979b0 (127.19.246.194:41885), d79e0bc2222843bea57748285363e322 (127.19.246.193:40799)
W20260810 03:24:13.313146 7394 tablet.cc:2369] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:13.314163 7396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.307321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:41885 (local address 127.19.246.195:44399)
0810 03:24:13.307682 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:13.307705 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:13.307867 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:13.308257 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:13.308262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:13.308290 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:13.308534 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.308547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.310099 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.310105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.311569 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.311584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.313315 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.313320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.313527 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.313552 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:24:13.313673 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 03:24:13.313847 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":3}
I20260810 03:24:13.314144 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.307466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:44399 (local address 127.19.246.194:41885)
0810 03:24:13.307688 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.307695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.307941 (+ 246us) server_negotiation.cc:407] Connection header received
0810 03:24:13.308037 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.308040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.308109 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.308222 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:13.308808 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.309933 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.311784 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.313138 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.313308 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.313393 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.313721 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.313873 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 03:24:13.315392 7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b10815d44a4d2bbc68bebfea897aed" candidate_uuid: "825f5f053688489198b7e8b19bfe3e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fc3332a64844f69799d09297d979b0" is_pre_election: true
I20260810 03:24:13.315834 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.307674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:35533 (local address 127.19.246.193:40799)
0810 03:24:13.307891 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.307897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.308504 (+ 607us) server_negotiation.cc:407] Connection header received
0810 03:24:13.308683 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.308691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.308783 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.308855 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:13.309628 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.310646 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.313286 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.314650 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.314875 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.314958 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.315404 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.315549 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:24:13.315881 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.307434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:40799 (local address 127.19.246.195:35533)
0810 03:24:13.308389 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:13.308405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:13.308528 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:13.308934 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:13.308940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:13.308969 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:13.309425 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.309438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.310852 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.310857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.313065 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:13.313090 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.315024 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.315032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:13.315207 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.315234 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:24:13.315348 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:24:13.315513 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":565,"threads_started":1}
I20260810 03:24:13.316087 7197 raft_consensus.cc:2454] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825f5f053688489198b7e8b19bfe3e74 in term 0.
I20260810 03:24:13.316985 7275 leader_election.cc:302] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 825f5f053688489198b7e8b19bfe3e74, e0fc3332a64844f69799d09297d979b0; no voters:
I20260810 03:24:13.317022 7055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b10815d44a4d2bbc68bebfea897aed" candidate_uuid: "825f5f053688489198b7e8b19bfe3e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e0bc2222843bea57748285363e322" is_pre_election: true
I20260810 03:24:13.317544 7424 raft_consensus.cc:2781] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:24:13.317586 7055 raft_consensus.cc:2454] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825f5f053688489198b7e8b19bfe3e74 in term 0.
I20260810 03:24:13.317728 7424 raft_consensus.cc:482] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:24:13.317878 7424 raft_consensus.cc:3037] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:13.320482 7424 raft_consensus.cc:504] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.321187 7424 leader_election.cc:288] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [CANDIDATE]: Term 1 election: Requested vote from peers e0fc3332a64844f69799d09297d979b0 (127.19.246.194:41885), d79e0bc2222843bea57748285363e322 (127.19.246.193:40799)
I20260810 03:24:13.321970 7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b10815d44a4d2bbc68bebfea897aed" candidate_uuid: "825f5f053688489198b7e8b19bfe3e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fc3332a64844f69799d09297d979b0"
I20260810 03:24:13.321982 7055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b10815d44a4d2bbc68bebfea897aed" candidate_uuid: "825f5f053688489198b7e8b19bfe3e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e0bc2222843bea57748285363e322"
I20260810 03:24:13.322256 7197 raft_consensus.cc:3037] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:13.322256 7055 raft_consensus.cc:3037] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:13.326321 7055 raft_consensus.cc:2454] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825f5f053688489198b7e8b19bfe3e74 in term 1.
I20260810 03:24:13.326321 7197 raft_consensus.cc:2454] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825f5f053688489198b7e8b19bfe3e74 in term 1.
I20260810 03:24:13.327006 7275 leader_election.cc:302] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 825f5f053688489198b7e8b19bfe3e74, e0fc3332a64844f69799d09297d979b0; no voters:
I20260810 03:24:13.327534 7424 raft_consensus.cc:2781] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:13.328150 7424 raft_consensus.cc:686] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 1 LEADER]: Becoming Leader. State: Replica: 825f5f053688489198b7e8b19bfe3e74, State: Running, Role: LEADER
I20260810 03:24:13.328697 7424 consensus_queue.cc:237] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } }
I20260810 03:24:13.334761 6895 catalog_manager.cc:5495] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 825f5f053688489198b7e8b19bfe3e74 (127.19.246.195). New cstate: current_term: 1 leader_uuid: "825f5f053688489198b7e8b19bfe3e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "825f5f053688489198b7e8b19bfe3e74" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 39461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 } health_report { overall_health: UNKNOWN } } }
I20260810 03:24:13.368171 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.362293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.19.246.195:39461)
0810 03:24:13.362501 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.362506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.362523 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:24:13.362629 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.362632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.362712 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.362858 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:24:13.363247 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.364058 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.365591 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.366004 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.366112 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.366723 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:24:13.367314 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:24:13.367623 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.367762 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.367840 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:24:13.377740 7197 raft_consensus.cc:1261] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 1 FOLLOWER]: Refusing update from remote peer 825f5f053688489198b7e8b19bfe3e74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:13.377772 7055 raft_consensus.cc:1261] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 1 FOLLOWER]: Refusing update from remote peer 825f5f053688489198b7e8b19bfe3e74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:13.378916 7426 consensus_queue.cc:1035] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fc3332a64844f69799d09297d979b0" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 41885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:13.379331 7424 consensus_queue.cc:1035] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79e0bc2222843bea57748285363e322" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 40799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:13.393180 7429 mvcc.cc:205] Tried to move back new op lower bound from 7316816909823578112 to 7316816909639245824. Current Snapshot: MvccSnapshot[applied={T|T < 7316816909823578112}]
I20260810 03:24:13.401208 7432 mvcc.cc:205] Tried to move back new op lower bound from 7316816909823578112 to 7316816909639245824. Current Snapshot: MvccSnapshot[applied={T|T < 7316816909823578112}]
I20260810 03:24:13.413235 6896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:13.413568 6896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:13.418730 6896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8162a69f6d374430b6660f30f031549f: Sending DeleteTablet for 3 replicas of tablet a7b10815d44a4d2bbc68bebfea897aed
I20260810 03:24:13.420501 7177 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b10815d44a4d2bbc68bebfea897aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:13 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:48024
I20260810 03:24:13.421468 7033 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b10815d44a4d2bbc68bebfea897aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:13 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:36284
I20260810 03:24:13.421787 7440 tablet_replica.cc:330] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: stopping tablet replica
I20260810 03:24:13.422315 7440 raft_consensus.cc:2229] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:13.422567 7324 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b10815d44a4d2bbc68bebfea897aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:13 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:39400
I20260810 03:24:13.423087 7440 raft_consensus.cc:2258] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:13.427284 7440 ts_tablet_manager.cc:1905] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:13.436735 7440 ts_tablet_manager.cc:1918] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:13.437048 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 6966
I20260810 03:24:13.437108 7440 log.cc:1196] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/a7b10815d44a4d2bbc68bebfea897aed
I20260810 03:24:13.437911 7440 ts_tablet_manager.cc:1939] T a7b10815d44a4d2bbc68bebfea897aed P e0fc3332a64844f69799d09297d979b0: Deleting consensus metadata
I20260810 03:24:13.440240 6881 catalog_manager.cc:4841] TS e0fc3332a64844f69799d09297d979b0 (127.19.246.194:41885): tablet a7b10815d44a4d2bbc68bebfea897aed (table test-table [id=ce70a1b2755e4cc1bd1ea97be93f2704]) successfully deleted
I20260810 03:24:13.441659 7442 tablet_replica.cc:330] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: stopping tablet replica
I20260810 03:24:13.442209 7442 raft_consensus.cc:2229] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:24:13.443773 7441 tablet_replica.cc:330] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: stopping tablet replica
I20260810 03:24:13.444306 7441 raft_consensus.cc:2229] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:13.443056 7442 raft_consensus.cc:2258] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:13.444922 7441 raft_consensus.cc:2258] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:13.447405 7441 ts_tablet_manager.cc:1905] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:13.447418 7442 ts_tablet_manager.cc:1905] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 03:24:13 dist-test-slave-btw8 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:24:13.455505 7441 ts_tablet_manager.cc:1918] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:13.455870 7441 log.cc:1196] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/a7b10815d44a4d2bbc68bebfea897aed
I20260810 03:24:13.456532 7441 ts_tablet_manager.cc:1939] T a7b10815d44a4d2bbc68bebfea897aed P d79e0bc2222843bea57748285363e322: Deleting consensus metadata
I20260810 03:24:13.457019 7442 ts_tablet_manager.cc:1918] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:13.457325 7442 log.cc:1196] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/a7b10815d44a4d2bbc68bebfea897aed
I20260810 03:24:13.458079 7442 ts_tablet_manager.cc:1939] T a7b10815d44a4d2bbc68bebfea897aed P 825f5f053688489198b7e8b19bfe3e74: Deleting consensus metadata
I20260810 03:24:13.458833 6880 catalog_manager.cc:4841] TS d79e0bc2222843bea57748285363e322 (127.19.246.193:40799): tablet a7b10815d44a4d2bbc68bebfea897aed (table test-table [id=ce70a1b2755e4cc1bd1ea97be93f2704]) successfully deleted
I20260810 03:24:13.460376 6881 catalog_manager.cc:4841] TS 825f5f053688489198b7e8b19bfe3e74 (127.19.246.195:39461): tablet a7b10815d44a4d2bbc68bebfea897aed (table test-table [id=ce70a1b2755e4cc1bd1ea97be93f2704]) successfully deleted
I20260810 03:24:13.462522 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.438747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.19.246.193:40799)
0810 03:24:13.438957 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.438965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.440666 (+ 1701us) server_negotiation.cc:407] Connection header received
0810 03:24:13.441012 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.441018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.441133 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.441251 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:13.443558 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.444832 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.447102 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.447768 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.447885 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.454620 (+ 6735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:13.454639 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:13.454660 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:13.454717 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:13.459192 (+ 4475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.460128 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.460140 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.460145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.460346 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.460814 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.460820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.460823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.461196 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:13.461528 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.461853 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.462137 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:24:13.463289 7098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:13.560348 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 6966
I20260810 03:24:13.587525 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 7110
Aug 10 03:24:13 dist-test-slave-btw8 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:24:13.608825 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.588580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.19.246.194:41885)
0810 03:24:13.588850 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.588857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.588878 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:24:13.588943 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.588947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.589010 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.589137 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:13.590532 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.591481 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.592821 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.593233 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.593374 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.602170 (+ 8796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:13.602189 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:13.602206 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:13.602257 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:13.605562 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.606448 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.606456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.606458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.606597 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.607022 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.607028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.607037 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.607537 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:13.607874 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.608222 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.608461 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:24:13.609628 7242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:13.704983 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 7110
I20260810 03:24:13.727954 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 7254
Aug 10 03:24:13 dist-test-slave-btw8 krb5kdc[6848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:24:13.751052 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:13.728902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.19.246.195:39461)
0810 03:24:13.729142 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:13.729150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:13.729314 (+ 164us) server_negotiation.cc:407] Connection header received
0810 03:24:13.729411 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:13.729417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:13.729510 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:13.729698 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:13.731376 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.732744 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.734244 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:13.735263 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:13.735621 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:13.742157 (+ 6536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:13.742175 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:13.742198 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:13.742257 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:13.747144 (+ 4887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.748329 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.748348 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.748352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.748659 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:13.749084 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:13.749092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:13.749097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:13.749704 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:13.750040 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:13.750328 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 03:24:13.750617 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:24:13.751962 7388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:13.883775 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 7254
I20260810 03:24:13.911670 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 6861
I20260810 03:24:13.912954 6931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:14.031884 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 6861
2026-08-10T03:24:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5178 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T03:24:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:14Z Disabled control of system clock
I20260810 03:24:14.142367 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:43029
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:43029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:24:14.525278 7451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:14.525655 7451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:14.525746 7451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:14.535075 7451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:14.535249 7451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:14.535295 7451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:14.535420 7451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:14.535498 7451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:14.546016 7451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:43029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:43029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:14.547545 7451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:14.549407 7451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:14.562958 7456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:14.562997 7459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:14.563040 7457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:14.564042 7451 server_base.cc:1034] running on GCE node
I20260810 03:24:14.564870 7451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:14.566388 7451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:14.567605 7451 hybrid_clock.cc:648] HybridClock initialized: now 1786332254567548 us; error 55 us; skew 500 ppm
I20260810 03:24:14.567996 7451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:14.570392 7451 webserver.cc:460] Webserver started at http://127.19.246.254:44669/ using document root <none> and password file <none>
I20260810 03:24:14.571105 7451 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:14.571208 7451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:14.571624 7451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:14.574803 7451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "527855c7c88e4466ac57c316019c41aa"
format_stamp: "Formatted at 2026-08-10 03:24:14 on dist-test-slave-btw8"
server_key: "74196969f3f4ae75d6ee98074969dc4e"
server_key_iv: "1e1f1455eb53838a5fed8ad8fe191ecb"
server_key_version: "encryptionkey@0"
I20260810 03:24:14.575762 7451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "527855c7c88e4466ac57c316019c41aa"
format_stamp: "Formatted at 2026-08-10 03:24:14 on dist-test-slave-btw8"
server_key: "74196969f3f4ae75d6ee98074969dc4e"
server_key_iv: "1e1f1455eb53838a5fed8ad8fe191ecb"
server_key_version: "encryptionkey@0"
I20260810 03:24:14.581758 7451 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:24:14.586107 7466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:14.587814 7451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:24:14.587989 7451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "527855c7c88e4466ac57c316019c41aa"
format_stamp: "Formatted at 2026-08-10 03:24:14 on dist-test-slave-btw8"
server_key: "74196969f3f4ae75d6ee98074969dc4e"
server_key_iv: "1e1f1455eb53838a5fed8ad8fe191ecb"
server_key_version: "encryptionkey@0"
I20260810 03:24:14.588228 7451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:14.633920 7451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:14.634931 7451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:14.635208 7451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:14.657776 7451 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:43029
I20260810 03:24:14.657807 7521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:43029 every 8 connection(s)
I20260810 03:24:14.659479 7451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:14.663542 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:14.669250 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 7451
I20260810 03:24:14.669473 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:14.669968 20443 external_mini_cluster.cc:1428] Setting key 5e334343d9de845ffcc4b22d6343f664
I20260810 03:24:14.676316 7522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa: Bootstrap starting.
I20260810 03:24:14.680653 7522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:14.681862 7522 log.cc:824] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:14.685856 7522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa: No bootstrap required, opened a new log
I20260810 03:24:14.687646 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:14.672307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.19.246.254:43029)
0810 03:24:14.673561 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:14.673577 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:14.673630 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:24:14.676160 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:14.676191 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:14.676855 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:14.677331 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:14.677984 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:14.680133 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:14.681771 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:14.682977 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:14.683128 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:14.684590 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:14.684601 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:14.684620 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:14.685396 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:14.685507 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:14.685533 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:14.685652 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:14.685919 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 03:24:14.686298 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":432,"threads_started":1}
I20260810 03:24:14.695530 7522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527855c7c88e4466ac57c316019c41aa" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43029 } }
I20260810 03:24:14.695933 7522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:14.696026 7522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527855c7c88e4466ac57c316019c41aa, State: Initialized, Role: FOLLOWER
I20260810 03:24:14.696631 7522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527855c7c88e4466ac57c316019c41aa" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43029 } }
I20260810 03:24:14.696933 7522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:14.697042 7522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:14.697190 7522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:14.699755 7522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527855c7c88e4466ac57c316019c41aa" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43029 } }
I20260810 03:24:14.700258 7522 leader_election.cc:302] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 527855c7c88e4466ac57c316019c41aa; no voters:
I20260810 03:24:14.700906 7522 leader_election.cc:288] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:14.701073 7527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:14.701881 7527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [term 1 LEADER]: Becoming Leader. State: Replica: 527855c7c88e4466ac57c316019c41aa, State: Running, Role: LEADER
I20260810 03:24:14.702467 7527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527855c7c88e4466ac57c316019c41aa" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43029 } }
I20260810 03:24:14.703258 7522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:14.705462 7529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527855c7c88e4466ac57c316019c41aa. Latest consensus state: current_term: 1 leader_uuid: "527855c7c88e4466ac57c316019c41aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527855c7c88e4466ac57c316019c41aa" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43029 } } }
I20260810 03:24:14.705901 7529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:14.710726 7528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "527855c7c88e4466ac57c316019c41aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527855c7c88e4466ac57c316019c41aa" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43029 } } }
I20260810 03:24:14.711043 7528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:14.715624 7532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:14.727419 7532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:14.740447 7532 catalog_manager.cc:1347] Generated new cluster ID: d7cfd83e306b4fbe93658577f561ba8c
I20260810 03:24:14.740610 7532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:14.753222 7532 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:14.754854 7532 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:14.769318 7532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa: Generated new TSK 0
I20260810 03:24:14.770432 7532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:14.776660 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43029
--builtin_ntp_servers=127.19.246.212:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:24:15.156898 7546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:15.157217 7546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:15.157311 7546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:15.165467 7546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:15.165619 7546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:15.165741 7546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:24:15.175812 7546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:15.177232 7546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:15.179315 7546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:15.197747 7551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:15.197731 7552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:15.197962 7554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:15.198731 7546 server_base.cc:1034] running on GCE node
I20260810 03:24:15.199837 7546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:15.201114 7546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:15.202394 7546 hybrid_clock.cc:648] HybridClock initialized: now 1786332255202342 us; error 48 us; skew 500 ppm
I20260810 03:24:15.202875 7546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:15.206056 7546 webserver.cc:460] Webserver started at http://127.19.246.193:44523/ using document root <none> and password file <none>
I20260810 03:24:15.206910 7546 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:15.207023 7546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:15.207509 7546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:15.210786 7546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "2f07dccd4fef4718b48c9fb73f08cd7b"
format_stamp: "Formatted at 2026-08-10 03:24:15 on dist-test-slave-btw8"
server_key: "04451ca0ea9820a556639ba50b4c1e93"
server_key_iv: "e6a647d195cc6cc9df73f5ec9a1d05af"
server_key_version: "encryptionkey@0"
I20260810 03:24:15.211853 7546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "2f07dccd4fef4718b48c9fb73f08cd7b"
format_stamp: "Formatted at 2026-08-10 03:24:15 on dist-test-slave-btw8"
server_key: "04451ca0ea9820a556639ba50b4c1e93"
server_key_iv: "e6a647d195cc6cc9df73f5ec9a1d05af"
server_key_version: "encryptionkey@0"
I20260810 03:24:15.217787 7546 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260810 03:24:15.221938 7561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:15.223598 7546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:24:15.223827 7546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "2f07dccd4fef4718b48c9fb73f08cd7b"
format_stamp: "Formatted at 2026-08-10 03:24:15 on dist-test-slave-btw8"
server_key: "04451ca0ea9820a556639ba50b4c1e93"
server_key_iv: "e6a647d195cc6cc9df73f5ec9a1d05af"
server_key_version: "encryptionkey@0"
I20260810 03:24:15.224082 7546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:15.257239 7546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:15.258498 7546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:15.258971 7546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:15.260257 7546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:15.262523 7546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:15.262641 7546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:15.262831 7546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:15.262923 7546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:15.303779 7546 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:37811
I20260810 03:24:15.303879 7682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:37811 every 8 connection(s)
I20260810 03:24:15.305403 7546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:24:15.313987 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 7546
I20260810 03:24:15.314204 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:24:15.314947 20443 external_mini_cluster.cc:1428] Setting key 2e6f368ac0b20a8f7c49b18f216634b9
I20260810 03:24:15.318535 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43029
--builtin_ntp_servers=127.19.246.212:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:24:15.321745 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:15.308460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:37659 (local address 127.19.246.254:43029)
0810 03:24:15.308993 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:15.308998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:15.310481 (+ 1483us) server_negotiation.cc:407] Connection header received
0810 03:24:15.312581 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:15.312587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:15.312719 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:15.312853 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:15.314422 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:15.315689 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:15.318020 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:15.318643 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:15.318831 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:15.320814 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:15.320820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:15.320824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:15.320844 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:15.320886 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:15.320899 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:15.321007 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:15.321268 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 03:24:15.321388 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":297,"threads_started":1}
I20260810 03:24:15.322914 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:15.309227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43029 (local address 127.19.246.193:37659)
0810 03:24:15.310289 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:15.310329 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:15.312240 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:15.313164 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:15.313196 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:15.313316 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:15.314006 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:15.314025 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:15.315934 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:15.315940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:15.317812 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:15.317829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:15.318982 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:15.318992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:15.319273 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:15.320236 (+ 963us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:15.320239 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:15.320298 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:15.320403 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:15.321010 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:15.321027 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:15.321037 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 03:24:15.321256 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 03:24:15.321732 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":355,"threads_started":1}
I20260810 03:24:15.324918 7683 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43029
I20260810 03:24:15.325302 7683 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:15.326150 7683 heartbeater.cc:511] Master 127.19.246.254:43029 requested a full tablet report, sending...
I20260810 03:24:15.328572 7485 ts_manager.cc:194] Registered new tserver with Master: 2f07dccd4fef4718b48c9fb73f08cd7b (127.19.246.193:37811)
I20260810 03:24:15.330489 7485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.193:37659
W20260810 03:24:15.709705 7687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:15.710070 7687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:15.710232 7687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:15.719211 7687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:15.719421 7687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:15.719619 7687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:24:15.731220 7687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:15.732790 7687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:15.734951 7687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:15.750021 7693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:15.750098 7694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:15.750303 7696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:15.756480 7687 server_base.cc:1034] running on GCE node
I20260810 03:24:15.757393 7687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:15.758759 7687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:15.760008 7687 hybrid_clock.cc:648] HybridClock initialized: now 1786332255759950 us; error 40 us; skew 500 ppm
I20260810 03:24:15.760419 7687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:15.763310 7687 webserver.cc:460] Webserver started at http://127.19.246.194:45915/ using document root <none> and password file <none>
I20260810 03:24:15.764251 7687 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:15.764401 7687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:15.764814 7687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:15.768538 7687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "0da98e25268549c1850f6551db8d54eb"
format_stamp: "Formatted at 2026-08-10 03:24:15 on dist-test-slave-btw8"
server_key: "89b2d3ce4e0f2afc759c723139353e34"
server_key_iv: "b64752e3c3aa5db31d8251281bcd1d07"
server_key_version: "encryptionkey@0"
I20260810 03:24:15.769654 7687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "0da98e25268549c1850f6551db8d54eb"
format_stamp: "Formatted at 2026-08-10 03:24:15 on dist-test-slave-btw8"
server_key: "89b2d3ce4e0f2afc759c723139353e34"
server_key_iv: "b64752e3c3aa5db31d8251281bcd1d07"
server_key_version: "encryptionkey@0"
I20260810 03:24:15.776508 7687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:24:15.781333 7703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:15.782914 7687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:24:15.783140 7687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "0da98e25268549c1850f6551db8d54eb"
format_stamp: "Formatted at 2026-08-10 03:24:15 on dist-test-slave-btw8"
server_key: "89b2d3ce4e0f2afc759c723139353e34"
server_key_iv: "b64752e3c3aa5db31d8251281bcd1d07"
server_key_version: "encryptionkey@0"
I20260810 03:24:15.783465 7687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:15.800563 7687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:15.801622 7687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:15.802016 7687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:15.803211 7687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:15.805271 7687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:15.805419 7687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:15.805588 7687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:15.805691 7687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:15.846709 7687 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:45413
I20260810 03:24:15.846760 7828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:45413 every 8 connection(s)
I20260810 03:24:15.848269 7687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:24:15.857901 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 7687
I20260810 03:24:15.858152 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:24:15.858628 20443 external_mini_cluster.cc:1428] Setting key a398f9e4642500d65fb6581b131f141e
I20260810 03:24:15.862119 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43029
--builtin_ntp_servers=127.19.246.212:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:24:15.863754 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:15.851214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:60235 (local address 127.19.246.254:43029)
0810 03:24:15.851822 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:15.851827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:15.853190 (+ 1363us) server_negotiation.cc:407] Connection header received
0810 03:24:15.855264 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:15.855269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:15.855414 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:15.855534 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:15.856890 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:15.857860 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:15.860170 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:15.860736 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:15.860840 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:15.862616 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:15.862625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:15.862633 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:15.862661 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:15.862722 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:15.862741 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:15.862847 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:15.863096 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 03:24:15.863237 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":284,"threads_started":1}
I20260810 03:24:15.864831 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:15.851894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43029 (local address 127.19.246.194:60235)
0810 03:24:15.853009 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:15.853051 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:15.854920 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:15.855891 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:15.855900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:15.856002 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:15.856711 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:15.856730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:15.858111 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:15.858118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:15.860003 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:15.860021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:15.861007 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:15.861020 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:15.861297 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:15.862142 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:15.862144 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:15.862232 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:15.862380 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:15.862862 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:15.862871 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:15.862882 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 03:24:15.863050 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 03:24:15.863482 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":370,"threads_started":1}
I20260810 03:24:15.866694 7830 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43029
I20260810 03:24:15.867061 7830 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:15.868129 7830 heartbeater.cc:511] Master 127.19.246.254:43029 requested a full tablet report, sending...
I20260810 03:24:15.870065 7485 ts_manager.cc:194] Registered new tserver with Master: 0da98e25268549c1850f6551db8d54eb (127.19.246.194:45413)
I20260810 03:24:15.871033 7485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.194:60235
W20260810 03:24:16.236394 7834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:16.236781 7834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:16.236949 7834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:16.246132 7834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:16.246297 7834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:16.246515 7834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:24:16.257115 7834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:16.258576 7834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:16.260581 7834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:16.276419 7839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:16.276356 7840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:16.276350 7842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:16.277160 7834 server_base.cc:1034] running on GCE node
I20260810 03:24:16.277870 7834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:16.279023 7834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:16.280332 7834 hybrid_clock.cc:648] HybridClock initialized: now 1786332256280253 us; error 94 us; skew 500 ppm
I20260810 03:24:16.280738 7834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:16.283409 7834 webserver.cc:460] Webserver started at http://127.19.246.195:37261/ using document root <none> and password file <none>
I20260810 03:24:16.284250 7834 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:16.284407 7834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:16.284770 7834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:16.288815 7834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "7ba678168c154dceb4d3967faeddec87"
format_stamp: "Formatted at 2026-08-10 03:24:16 on dist-test-slave-btw8"
server_key: "917390b3c925c879d804f7e2b4969fee"
server_key_iv: "813cb8e3ee069ee26438cb8f68bf061c"
server_key_version: "encryptionkey@0"
I20260810 03:24:16.290014 7834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "7ba678168c154dceb4d3967faeddec87"
format_stamp: "Formatted at 2026-08-10 03:24:16 on dist-test-slave-btw8"
server_key: "917390b3c925c879d804f7e2b4969fee"
server_key_iv: "813cb8e3ee069ee26438cb8f68bf061c"
server_key_version: "encryptionkey@0"
I20260810 03:24:16.297312 7834 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260810 03:24:16.301905 7848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:16.303746 7834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:24:16.303956 7834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "7ba678168c154dceb4d3967faeddec87"
format_stamp: "Formatted at 2026-08-10 03:24:16 on dist-test-slave-btw8"
server_key: "917390b3c925c879d804f7e2b4969fee"
server_key_iv: "813cb8e3ee069ee26438cb8f68bf061c"
server_key_version: "encryptionkey@0"
I20260810 03:24:16.304270 7834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:16.334090 7683 heartbeater.cc:503] Master 127.19.246.254:43029 was elected leader, sending a full tablet report...
I20260810 03:24:16.338812 7834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:16.340061 7834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:16.340478 7834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:16.341934 7834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:16.343806 7834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:16.343910 7834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:16.344087 7834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:16.344163 7834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:16.382323 7834 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:41155
I20260810 03:24:16.382453 7970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:41155 every 8 connection(s)
I20260810 03:24:16.384029 7834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:24:16.391650 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 7834
I20260810 03:24:16.391907 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:24:16.392567 20443 external_mini_cluster.cc:1428] Setting key bb59ba99e30fe253f22eddc89ebcb5c4
I20260810 03:24:16.400331 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.386900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:60241 (local address 127.19.246.254:43029)
0810 03:24:16.387483 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.387488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.389116 (+ 1628us) server_negotiation.cc:407] Connection header received
0810 03:24:16.391298 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.391303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.391469 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.391611 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:16.393462 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.394779 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.397026 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.397464 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.397610 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.399480 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:16.399487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:16.399490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:16.399512 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:16.399557 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:16.399570 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:16.399652 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.399911 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.400016 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":324,"threads_started":1}
I20260810 03:24:16.401561 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.387496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43029 (local address 127.19.246.195:60241)
0810 03:24:16.388872 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:16.388917 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:16.390866 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:16.391968 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:16.391985 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:16.392136 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:16.393211 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.393238 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.395020 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.395031 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.396860 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.396875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.397916 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.397921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.398161 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.399022 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:16.399025 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:16.399072 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:16.399167 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:16.399692 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:16.399701 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:16.399708 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:24:16.399877 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 03:24:16.400222 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":329,"threads_started":1}
I20260810 03:24:16.403617 7971 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43029
I20260810 03:24:16.404039 7971 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:16.405143 7971 heartbeater.cc:511] Master 127.19.246.254:43029 requested a full tablet report, sending...
I20260810 03:24:16.407469 7485 ts_manager.cc:194] Registered new tserver with Master: 7ba678168c154dceb4d3967faeddec87 (127.19.246.195:41155)
I20260810 03:24:16.408691 7485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.195:60241
I20260810 03:24:16.410101 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:24:16.423544 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.416085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.19.246.254:43029)
0810 03:24:16.416407 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.416415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.416745 (+ 330us) server_negotiation.cc:407] Connection header received
0810 03:24:16.416979 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.416984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.417059 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.417158 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:16.417849 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.418598 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.420377 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.420849 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.420982 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.422442 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:16.422455 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:16.422458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:16.422485 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:16.422524 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:16.422537 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:16.422618 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.422865 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.422960 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260810 03:24:16.428501 7485 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:24:16.437227 7485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:24:16.467926 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.452404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.19.246.194:45413)
0810 03:24:16.456494 (+ 4090us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.456507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.456550 (+ 43us) server_negotiation.cc:407] Connection header received
0810 03:24:16.456695 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.456707 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.457064 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.458346 (+ 1282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:16.458638 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.460576 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.464366 (+ 3790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.466298 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.466553 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.466794 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.467106 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.467332 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3872,"thread_start_us":2344,"threads_started":1}
I20260810 03:24:16.467896 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.454681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:37811 (local address 127.0.0.1:42184)
0810 03:24:16.458562 (+ 3881us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:16.458588 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:16.458728 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:16.460103 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:16.460108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:16.460142 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:16.460588 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.460600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.462161 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.462166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.464352 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.464366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.466063 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.466073 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.466459 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.466555 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 03:24:16.466728 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 03:24:16.466948 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3716,"mutex_wait_us":127,"thread_start_us":1239,"threads_started":1}
I20260810 03:24:16.468390 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.452275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:41155 (local address 127.0.0.1:58912)
0810 03:24:16.456345 (+ 4070us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:16.456375 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:16.456598 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:16.458892 (+ 2294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:16.458896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:16.458925 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:16.459341 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.459382 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.461415 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.461422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.464368 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.464384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.467117 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.467126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.467682 (+ 556us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.467723 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:24:16.467829 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:24:16.468026 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3856,"mutex_wait_us":114,"thread_start_us":1483,"threads_started":1}
I20260810 03:24:16.467953 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.452069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:45413 (local address 127.0.0.1:37692)
0810 03:24:16.454220 (+ 2151us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:16.454262 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:16.454451 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:16.457389 (+ 2938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:16.457402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:16.457447 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:16.458065 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.458083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.460743 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.460749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.463988 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.464012 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.466700 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.466709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.466872 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.466903 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:24:16.466997 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:24:16.467127 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"mutex_wait_us":145,"thread_start_us":6133,"threads_started":1}
I20260810 03:24:16.469532 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.455345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.19.246.193:37811)
0810 03:24:16.459022 (+ 3677us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.459033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.459073 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:24:16.459196 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.459208 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.459565 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.459797 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:16.460804 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.462001 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.464632 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.465828 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.466092 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.466508 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.468931 (+ 2423us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.469164 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3406,"thread_start_us":3226,"threads_started":1}
I20260810 03:24:16.470465 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.452673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.19.246.195:41155)
0810 03:24:16.455667 (+ 2994us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.455683 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.456814 (+ 1131us) server_negotiation.cc:407] Connection header received
0810 03:24:16.456983 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.456994 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.457336 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.457628 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:16.459618 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.461246 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.464704 (+ 3458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.466443 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.466685 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.466905 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.468282 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.468484 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2721,"thread_start_us":2506,"threads_started":1}
I20260810 03:24:16.473273 7614 tablet_service.cc:1467] Processing CreateTablet for tablet 26045216333b48f7b9a44c6d0caeab89 (DEFAULT_TABLE table=test-table [id=48d6be8b5daf42aa88976f764c3c43da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:16.473205 7757 tablet_service.cc:1467] Processing CreateTablet for tablet 26045216333b48f7b9a44c6d0caeab89 (DEFAULT_TABLE table=test-table [id=48d6be8b5daf42aa88976f764c3c43da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:16.474647 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26045216333b48f7b9a44c6d0caeab89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:16.474704 7757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26045216333b48f7b9a44c6d0caeab89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:16.474411 7902 tablet_service.cc:1467] Processing CreateTablet for tablet 26045216333b48f7b9a44c6d0caeab89 (DEFAULT_TABLE table=test-table [id=48d6be8b5daf42aa88976f764c3c43da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:16.476007 7902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26045216333b48f7b9a44c6d0caeab89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:16.487910 7991 tablet_bootstrap.cc:492] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Bootstrap starting.
I20260810 03:24:16.491824 7992 tablet_bootstrap.cc:492] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Bootstrap starting.
I20260810 03:24:16.492188 7991 tablet_bootstrap.cc:654] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:16.494943 7991 log.cc:824] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:16.496263 7992 tablet_bootstrap.cc:654] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:16.496335 7993 tablet_bootstrap.cc:492] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Bootstrap starting.
I20260810 03:24:16.497933 7992 log.cc:824] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:16.500470 7993 tablet_bootstrap.cc:654] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:16.501883 7992 tablet_bootstrap.cc:492] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: No bootstrap required, opened a new log
I20260810 03:24:16.501993 7993 log.cc:824] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:16.502104 7991 tablet_bootstrap.cc:492] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: No bootstrap required, opened a new log
I20260810 03:24:16.502265 7992 ts_tablet_manager.cc:1397] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260810 03:24:16.502388 7991 ts_tablet_manager.cc:1397] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260810 03:24:16.505796 7993 tablet_bootstrap.cc:492] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: No bootstrap required, opened a new log
I20260810 03:24:16.506229 7993 ts_tablet_manager.cc:1397] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260810 03:24:16.511991 7991 raft_consensus.cc:348] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.512475 7991 raft_consensus.cc:374] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:16.512574 7991 raft_consensus.cc:729] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f07dccd4fef4718b48c9fb73f08cd7b, State: Initialized, Role: FOLLOWER
I20260810 03:24:16.513343 7991 consensus_queue.cc:260] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.514056 7992 raft_consensus.cc:348] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.514571 7992 raft_consensus.cc:374] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:16.514680 7992 raft_consensus.cc:729] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da98e25268549c1850f6551db8d54eb, State: Initialized, Role: FOLLOWER
I20260810 03:24:16.515614 7991 ts_tablet_manager.cc:1428] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Time spent starting tablet: real 0.013s user 0.015s sys 0.001s
I20260810 03:24:16.515573 7992 consensus_queue.cc:260] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.518718 7993 raft_consensus.cc:348] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.519474 7993 raft_consensus.cc:374] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:16.519626 7993 raft_consensus.cc:729] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba678168c154dceb4d3967faeddec87, State: Initialized, Role: FOLLOWER
I20260810 03:24:16.520489 7993 consensus_queue.cc:260] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.522320 7971 heartbeater.cc:503] Master 127.19.246.254:43029 was elected leader, sending a full tablet report...
I20260810 03:24:16.522907 7993 ts_tablet_manager.cc:1428] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260810 03:24:16.523797 7830 heartbeater.cc:503] Master 127.19.246.254:43029 was elected leader, sending a full tablet report...
I20260810 03:24:16.523945 7992 ts_tablet_manager.cc:1428] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Time spent starting tablet: real 0.022s user 0.018s sys 0.004s
W20260810 03:24:16.558303 7684 tablet.cc:2369] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:24:16.600852 7831 tablet.cc:2369] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:16.631438 7999 raft_consensus.cc:482] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:24:16.631884 7999 raft_consensus.cc:504] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.633911 7999 leader_election.cc:288] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0da98e25268549c1850f6551db8d54eb (127.19.246.194:45413), 2f07dccd4fef4718b48c9fb73f08cd7b (127.19.246.193:37811)
W20260810 03:24:16.636052 7972 tablet.cc:2369] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:16.640766 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.634291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:45413 (local address 127.19.246.195:38311)
0810 03:24:16.634649 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:16.634668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:16.634838 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:16.635439 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:16.635447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:16.635483 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:16.635657 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.635665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.637229 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.637236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.638793 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.638810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.639957 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.639966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.640141 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.640166 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:24:16.640298 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 03:24:16.640457 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260810 03:24:16.640765 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.634430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:38311 (local address 127.19.246.194:45413)
0810 03:24:16.634997 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.635003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.635023 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:24:16.635107 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.635113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.635218 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.635341 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:16.636315 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.637084 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.638975 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.639799 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.639976 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.640061 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.640328 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.640460 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397}
I20260810 03:24:16.641103 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.634466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:55943 (local address 127.19.246.193:37811)
0810 03:24:16.634773 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.634783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.635323 (+ 540us) server_negotiation.cc:407] Connection header received
0810 03:24:16.635530 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.635539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.635624 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.635777 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:16.636438 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.637327 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.639282 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.640191 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.640374 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.640444 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.640739 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.640872 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 03:24:16.641163 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.634429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:37811 (local address 127.19.246.195:55943)
0810 03:24:16.635208 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:16.635222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:16.635355 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:16.635816 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:16.635820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:16.635839 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:16.636240 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.636254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.637526 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.637534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.639055 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:16.639069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.640374 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.640381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:16.640561 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.640587 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:24:16.640701 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:24:16.640831 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":461,"threads_started":1}
I20260810 03:24:16.642238 7636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26045216333b48f7b9a44c6d0caeab89" candidate_uuid: "7ba678168c154dceb4d3967faeddec87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" is_pre_election: true
I20260810 03:24:16.642278 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26045216333b48f7b9a44c6d0caeab89" candidate_uuid: "7ba678168c154dceb4d3967faeddec87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da98e25268549c1850f6551db8d54eb" is_pre_election: true
I20260810 03:24:16.642935 7636 raft_consensus.cc:2454] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ba678168c154dceb4d3967faeddec87 in term 0.
I20260810 03:24:16.642935 7779 raft_consensus.cc:2454] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ba678168c154dceb4d3967faeddec87 in term 0.
I20260810 03:24:16.643796 7850 leader_election.cc:302] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f07dccd4fef4718b48c9fb73f08cd7b, 7ba678168c154dceb4d3967faeddec87; no voters:
I20260810 03:24:16.644376 7999 raft_consensus.cc:2781] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:24:16.644642 7999 raft_consensus.cc:482] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:24:16.644784 7999 raft_consensus.cc:3037] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:16.647516 7999 raft_consensus.cc:504] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.648432 7999 leader_election.cc:288] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [CANDIDATE]: Term 1 election: Requested vote from peers 0da98e25268549c1850f6551db8d54eb (127.19.246.194:45413), 2f07dccd4fef4718b48c9fb73f08cd7b (127.19.246.193:37811)
I20260810 03:24:16.649076 7636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26045216333b48f7b9a44c6d0caeab89" candidate_uuid: "7ba678168c154dceb4d3967faeddec87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b"
I20260810 03:24:16.649097 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26045216333b48f7b9a44c6d0caeab89" candidate_uuid: "7ba678168c154dceb4d3967faeddec87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0da98e25268549c1850f6551db8d54eb"
I20260810 03:24:16.649358 7636 raft_consensus.cc:3037] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:16.649396 7779 raft_consensus.cc:3037] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:16.652009 7779 raft_consensus.cc:2454] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ba678168c154dceb4d3967faeddec87 in term 1.
I20260810 03:24:16.652225 7636 raft_consensus.cc:2454] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ba678168c154dceb4d3967faeddec87 in term 1.
I20260810 03:24:16.652572 7849 leader_election.cc:302] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da98e25268549c1850f6551db8d54eb, 7ba678168c154dceb4d3967faeddec87; no voters:
I20260810 03:24:16.653033 7999 raft_consensus.cc:2781] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:16.653584 7999 raft_consensus.cc:686] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 1 LEADER]: Becoming Leader. State: Replica: 7ba678168c154dceb4d3967faeddec87, State: Running, Role: LEADER
I20260810 03:24:16.654196 7999 consensus_queue.cc:237] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } }
I20260810 03:24:16.659950 7485 catalog_manager.cc:5495] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ba678168c154dceb4d3967faeddec87 (127.19.246.195). New cstate: current_term: 1 leader_uuid: "7ba678168c154dceb4d3967faeddec87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ba678168c154dceb4d3967faeddec87" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 } health_report { overall_health: UNKNOWN } } }
I20260810 03:24:16.709021 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.702500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.19.246.195:41155)
0810 03:24:16.702743 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.702750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.702775 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:24:16.703029 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.703044 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.703141 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.703297 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:24:16.703984 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.705037 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.706575 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.707012 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.707112 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.707820 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:24:16.708491 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:24:16.708668 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.708744 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.708819 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:24:16.718601 7636 raft_consensus.cc:1261] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 1 FOLLOWER]: Refusing update from remote peer 7ba678168c154dceb4d3967faeddec87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:16.718601 7779 raft_consensus.cc:1261] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 1 FOLLOWER]: Refusing update from remote peer 7ba678168c154dceb4d3967faeddec87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:16.719830 7999 consensus_queue.cc:1035] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f07dccd4fef4718b48c9fb73f08cd7b" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:16.720213 8001 consensus_queue.cc:1035] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da98e25268549c1850f6551db8d54eb" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:16.744092 8005 mvcc.cc:205] Tried to move back new op lower bound from 7316816923507728384 to 7316816923260014592. Current Snapshot: MvccSnapshot[applied={T|T < 7316816923507728384}]
I20260810 03:24:16.759505 7485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:16.759797 7485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:16.764324 7485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 527855c7c88e4466ac57c316019c41aa: Sending DeleteTablet for 3 replicas of tablet 26045216333b48f7b9a44c6d0caeab89
I20260810 03:24:16.766014 7757 tablet_service.cc:1514] Processing DeleteTablet for tablet 26045216333b48f7b9a44c6d0caeab89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:16 UTC) from {username='slave'} at 127.0.0.1:37692
I20260810 03:24:16.766000 7902 tablet_service.cc:1514] Processing DeleteTablet for tablet 26045216333b48f7b9a44c6d0caeab89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:16 UTC) from {username='slave'} at 127.0.0.1:58912
I20260810 03:24:16.766162 7614 tablet_service.cc:1514] Processing DeleteTablet for tablet 26045216333b48f7b9a44c6d0caeab89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:16 UTC) from {username='slave'} at 127.0.0.1:42184
I20260810 03:24:16.767175 8017 tablet_replica.cc:330] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: stopping tablet replica
I20260810 03:24:16.767721 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 7546
I20260810 03:24:16.767760 8017 raft_consensus.cc:2229] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:16.768276 8017 raft_consensus.cc:2258] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:16.769186 8015 tablet_replica.cc:330] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: stopping tablet replica
I20260810 03:24:16.769630 8015 raft_consensus.cc:2229] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:16.770299 8015 raft_consensus.cc:2258] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:16.771410 8016 tablet_replica.cc:330] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: stopping tablet replica
I20260810 03:24:16.771994 8016 raft_consensus.cc:2229] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:24:16.773015 8016 raft_consensus.cc:2258] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:16.773265 8015 ts_tablet_manager.cc:1905] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:16.775801 8017 ts_tablet_manager.cc:1905] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:16.781479 8015 ts_tablet_manager.cc:1918] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:16.781718 8015 log.cc:1196] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/26045216333b48f7b9a44c6d0caeab89
I20260810 03:24:16.782332 8015 ts_tablet_manager.cc:1939] T 26045216333b48f7b9a44c6d0caeab89 P 0da98e25268549c1850f6551db8d54eb: Deleting consensus metadata
I20260810 03:24:16.785388 7467 catalog_manager.cc:4841] TS 0da98e25268549c1850f6551db8d54eb (127.19.246.194:45413): tablet 26045216333b48f7b9a44c6d0caeab89 (table test-table [id=48d6be8b5daf42aa88976f764c3c43da]) successfully deleted
I20260810 03:24:16.785643 8016 ts_tablet_manager.cc:1905] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:16.788393 8017 ts_tablet_manager.cc:1918] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:16.788719 8017 log.cc:1196] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/26045216333b48f7b9a44c6d0caeab89
I20260810 03:24:16.789518 8017 ts_tablet_manager.cc:1939] T 26045216333b48f7b9a44c6d0caeab89 P 2f07dccd4fef4718b48c9fb73f08cd7b: Deleting consensus metadata
I20260810 03:24:16.792295 7468 catalog_manager.cc:4841] TS 2f07dccd4fef4718b48c9fb73f08cd7b (127.19.246.193:37811): tablet 26045216333b48f7b9a44c6d0caeab89 (table test-table [id=48d6be8b5daf42aa88976f764c3c43da]) successfully deleted
I20260810 03:24:16.793326 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.769296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.19.246.193:37811)
0810 03:24:16.769475 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.769482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.784009 (+ 14527us) server_negotiation.cc:407] Connection header received
0810 03:24:16.784214 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.784229 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.784366 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.784479 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:16.785209 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.786324 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.788449 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.789109 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.789239 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.790815 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:16.790828 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:16.790846 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:16.791785 (+ 939us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:16.791853 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:16.791879 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:16.791991 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.792793 (+ 802us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.792923 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:24:16.793771 8016 ts_tablet_manager.cc:1918] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:16.794025 8016 log.cc:1196] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/26045216333b48f7b9a44c6d0caeab89
I20260810 03:24:16.794126 7678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:16.794665 8016 ts_tablet_manager.cc:1939] T 26045216333b48f7b9a44c6d0caeab89 P 7ba678168c154dceb4d3967faeddec87: Deleting consensus metadata
I20260810 03:24:16.796792 7468 catalog_manager.cc:4841] TS 7ba678168c154dceb4d3967faeddec87 (127.19.246.195:41155): tablet 26045216333b48f7b9a44c6d0caeab89 (table test-table [id=48d6be8b5daf42aa88976f764c3c43da]) successfully deleted
I20260810 03:24:16.888079 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 7546
I20260810 03:24:16.910856 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 7687
I20260810 03:24:16.920805 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:16.912033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.19.246.194:45413)
0810 03:24:16.912257 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:16.912263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:16.912433 (+ 170us) server_negotiation.cc:407] Connection header received
0810 03:24:16.912664 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:16.912670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:16.912808 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:16.912927 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:16.913564 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.914440 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.916725 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:16.917241 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:16.917370 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:16.919032 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:16.919043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:16.919059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:16.919852 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:16.919909 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:16.919941 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:16.920043 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:16.920265 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 03:24:16.920380 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 03:24:16.921666 7824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:17.019323 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 7687
I20260810 03:24:17.040505 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 7834
I20260810 03:24:17.048221 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:17.041440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.19.246.195:41155)
0810 03:24:17.041663 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:17.041670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:17.041709 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:24:17.041841 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:17.041845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:17.041932 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:17.042057 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:17.042506 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:17.043344 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:17.044742 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:17.045214 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:17.045357 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:17.046554 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:17.046564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:17.046580 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:17.047164 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:17.047220 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:17.047242 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:17.047400 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:17.047718 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:24:17.047830 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:24:17.049119 7966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:17.178428 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 7834
I20260810 03:24:17.202397 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 7451
I20260810 03:24:17.203758 7517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:17.318557 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 7451
2026-08-10T03:24:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3284 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:17 dist-test-slave-btw8 krb5kdc[8026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:17 dist-test-slave-btw8 krb5kdc[8026](info): set up 2 sockets
Aug 10 03:24:17 dist-test-slave-btw8 krb5kdc[8026](info): commencing operation
krb5kdc: starting...
W20260810 03:24:18.556692 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.005s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:18 dist-test-slave-btw8 krb5kdc[8026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:24:18.949265 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:33257
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:33257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:19.341287 8040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:19.341657 8040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:19.341758 8040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:19.349674 8040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:19.349826 8040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:19.349876 8040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:19.349928 8040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:19.349974 8040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:19.360918 8040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:33257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:19.362368 8040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:19.363770 8040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:19.378542 8045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:19.378810 8049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:19.378865 8046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:19.380059 8040 server_base.cc:1034] running on GCE node
I20260810 03:24:19.381109 8040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:19.382782 8040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:19.384210 8040 hybrid_clock.cc:648] HybridClock initialized: now 1786332259384133 us; error 52 us; skew 500 ppm
I20260810 03:24:19.384912 8040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:19 dist-test-slave-btw8 krb5kdc[8026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332259, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:19.388754 8040 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:24:19.390635 8040 webserver.cc:460] Webserver started at http://127.19.246.254:33355/ using document root <none> and password file <none>
I20260810 03:24:19.391615 8040 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:19.391780 8040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:19.392154 8040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:19.395789 8040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "aff3ef6ba9b64589aaccafe273f83df3"
format_stamp: "Formatted at 2026-08-10 03:24:19 on dist-test-slave-btw8"
server_key: "857b4669f71831d0cdaaa67a5c92a7ff"
server_key_iv: "e1c05997c7c9a3918e8ae403ee7ef730"
server_key_version: "encryptionkey@0"
I20260810 03:24:19.396687 8040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "aff3ef6ba9b64589aaccafe273f83df3"
format_stamp: "Formatted at 2026-08-10 03:24:19 on dist-test-slave-btw8"
server_key: "857b4669f71831d0cdaaa67a5c92a7ff"
server_key_iv: "e1c05997c7c9a3918e8ae403ee7ef730"
server_key_version: "encryptionkey@0"
I20260810 03:24:19.402995 8040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:24:19.408238 8056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:19.409953 8040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:24:19.410234 8040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "aff3ef6ba9b64589aaccafe273f83df3"
format_stamp: "Formatted at 2026-08-10 03:24:19 on dist-test-slave-btw8"
server_key: "857b4669f71831d0cdaaa67a5c92a7ff"
server_key_iv: "e1c05997c7c9a3918e8ae403ee7ef730"
server_key_version: "encryptionkey@0"
I20260810 03:24:19.410492 8040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:19.450750 8040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:19.456594 8040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:19.456972 8040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:19.480453 8040 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:33257
I20260810 03:24:19.480486 8114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:33257 every 8 connection(s)
I20260810 03:24:19.482559 8040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:19.487779 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 8040
I20260810 03:24:19.488051 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:19.488049 8115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:19.488775 20443 external_mini_cluster.cc:1428] Setting key af516c43dd321bfae7808c5076b88dd5
I20260810 03:24:19.505546 8115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3: Bootstrap starting.
Aug 10 03:24:19 dist-test-slave-btw8 krb5kdc[8026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:19.510151 8115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:19.511799 8115 log.cc:824] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:19.515537 8115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3: No bootstrap required, opened a new log
I20260810 03:24:19.517412 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:19.491396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.19.246.254:33257)
0810 03:24:19.492428 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:19.492442 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:19.492489 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:24:19.494626 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:19.494655 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:19.495184 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:19.495800 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:19.497610 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:19.499005 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:19.500673 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:19.501501 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:19.501761 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:19.507560 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:19.507581 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:19.507613 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:19.507677 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:19.512043 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:19.513397 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:19.513406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:19.513413 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:19.513618 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:19.514075 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:19.514079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:19.514081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:19.514670 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:19.515053 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:19.515614 (+ 561us) server_negotiation.cc:299] Negotiation successful
0810 03:24:19.516106 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":468,"threads_started":1}
I20260810 03:24:19.525986 8115 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3ef6ba9b64589aaccafe273f83df3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33257 } }
I20260810 03:24:19.526499 8115 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:19.526649 8115 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff3ef6ba9b64589aaccafe273f83df3, State: Initialized, Role: FOLLOWER
I20260810 03:24:19.527597 8115 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3ef6ba9b64589aaccafe273f83df3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33257 } }
I20260810 03:24:19.527946 8115 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:19.528141 8115 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:19.528374 8115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:19.531013 8115 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3ef6ba9b64589aaccafe273f83df3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33257 } }
I20260810 03:24:19.531761 8115 leader_election.cc:302] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff3ef6ba9b64589aaccafe273f83df3; no voters:
I20260810 03:24:19.532624 8115 leader_election.cc:288] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:19.532791 8120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:19.533658 8120 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [term 1 LEADER]: Becoming Leader. State: Replica: aff3ef6ba9b64589aaccafe273f83df3, State: Running, Role: LEADER
I20260810 03:24:19.534505 8120 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3ef6ba9b64589aaccafe273f83df3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33257 } }
I20260810 03:24:19.535724 8115 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:19.538383 8121 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff3ef6ba9b64589aaccafe273f83df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3ef6ba9b64589aaccafe273f83df3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33257 } } }
I20260810 03:24:19.538744 8122 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff3ef6ba9b64589aaccafe273f83df3. Latest consensus state: current_term: 1 leader_uuid: "aff3ef6ba9b64589aaccafe273f83df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff3ef6ba9b64589aaccafe273f83df3" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33257 } } }
I20260810 03:24:19.539022 8121 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:19.539083 8122 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:19.540133 8126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:19.546588 8126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:19.558902 8126 catalog_manager.cc:1347] Generated new cluster ID: 803b19eb16e54ef2ae751f23cdcda6bc
I20260810 03:24:19.559080 8126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:19.585837 8126 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:19.587517 8126 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:19.599042 8126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3: Generated new TSK 0
I20260810 03:24:19.600195 8126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:24:19.695024 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:33257
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:20.094007 8143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:20.094415 8143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:20.094596 8143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:20.102671 8143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:20.102859 8143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:20.102988 8143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:24:20.114508 8143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:20.116165 8143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:20.117661 8143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:20.133728 8151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:20.133770 8148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:20.133750 8149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:20.134930 8143 server_base.cc:1034] running on GCE node
I20260810 03:24:20.135780 8143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:20.137045 8143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:20.138411 8143 hybrid_clock.cc:648] HybridClock initialized: now 1786332260138335 us; error 70 us; skew 500 ppm
I20260810 03:24:20.138871 8143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:20 dist-test-slave-btw8 krb5kdc[8026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:20.142793 8143 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:24:20.144819 8143 webserver.cc:460] Webserver started at http://127.19.246.193:34379/ using document root <none> and password file <none>
I20260810 03:24:20.145571 8143 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:20.145752 8143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:20.146103 8143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:20.149444 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "ee6e96cca73e459b8a5ca698ea4f1611"
format_stamp: "Formatted at 2026-08-10 03:24:20 on dist-test-slave-btw8"
server_key: "19732a996a12e360e2e2b0c81af3cdbe"
server_key_iv: "50046b1d1878499670c71e536e094a5d"
server_key_version: "encryptionkey@0"
I20260810 03:24:20.150336 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "ee6e96cca73e459b8a5ca698ea4f1611"
format_stamp: "Formatted at 2026-08-10 03:24:20 on dist-test-slave-btw8"
server_key: "19732a996a12e360e2e2b0c81af3cdbe"
server_key_iv: "50046b1d1878499670c71e536e094a5d"
server_key_version: "encryptionkey@0"
I20260810 03:24:20.156515 8143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 03:24:20.160758 8159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:20.162506 8143 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:24:20.162817 8143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "ee6e96cca73e459b8a5ca698ea4f1611"
format_stamp: "Formatted at 2026-08-10 03:24:20 on dist-test-slave-btw8"
server_key: "19732a996a12e360e2e2b0c81af3cdbe"
server_key_iv: "50046b1d1878499670c71e536e094a5d"
server_key_version: "encryptionkey@0"
I20260810 03:24:20.163079 8143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:20.190136 8143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:20.195029 8143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:20.195427 8143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:20.196599 8143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:20.198025 8143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:20.198132 8143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:20.198294 8143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:20.198378 8143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:20.237473 8143 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:39873
I20260810 03:24:20.237520 8281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:39873 every 8 connection(s)
I20260810 03:24:20.238943 8143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:24:20.246264 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 8143
I20260810 03:24:20.246484 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:24:20.247064 20443 external_mini_cluster.cc:1428] Setting key 335900b34038c94ac8c89ae230d9e794
Aug 10 03:24:20 dist-test-slave-btw8 krb5kdc[8026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:20.263947 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:20.241910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:41837 (local address 127.19.246.254:33257)
0810 03:24:20.242570 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:20.242576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:20.243532 (+ 956us) server_negotiation.cc:407] Connection header received
0810 03:24:20.245614 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:20.245620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:20.245723 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:20.245861 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:20.247932 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:20.248698 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:20.250606 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:20.251453 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:20.251620 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:20.255902 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:20.255912 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:20.255919 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:20.255970 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:20.259959 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:20.260961 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:20.260970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:20.260975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:20.261155 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:20.261628 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:20.261636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:20.261642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:20.262093 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:20.262264 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:20.263079 (+ 815us) server_negotiation.cc:299] Negotiation successful
0810 03:24:20.263418 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":443,"threads_started":1}
I20260810 03:24:20.264807 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:20.242436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:33257 (local address 127.19.246.193:41837)
0810 03:24:20.243307 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:20.243374 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:20.245288 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:20.246134 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:20.246143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:20.246855 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:20.247691 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:20.247737 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:20.248891 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:20.248896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:20.250408 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:20.250423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:20.251694 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:20.251699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:20.251911 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:20.252792 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:20.252819 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:20.255607 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:20.260183 (+ 4576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:20.260192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:20.260234 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:20.260791 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:20.261278 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:20.261282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:20.261284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:20.261495 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:20.262241 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:20.262248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:20.262625 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 03:24:20.262887 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 03:24:20.263453 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":304,"threads_started":1}
I20260810 03:24:20.267207 8282 heartbeater.cc:347] Connected to a master server at 127.19.246.254:33257
I20260810 03:24:20.267714 8282 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:20.268836 8282 heartbeater.cc:511] Master 127.19.246.254:33257 requested a full tablet report, sending...
I20260810 03:24:20.272043 8077 ts_manager.cc:194] Registered new tserver with Master: ee6e96cca73e459b8a5ca698ea4f1611 (127.19.246.193:39873)
I20260810 03:24:20.274471 8077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:41837
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:24:20.332383 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:33257
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:20.710857 8291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:20.711179 8291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:20.711308 8291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:20.719427 8291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:20.719559 8291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:20.719686 8291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:24:20.730281 8291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:20.731745 8291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:20.733552 8291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:20.749948 8298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:20.750056 8297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:20.750119 8300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:20.751168 8291 server_base.cc:1034] running on GCE node
I20260810 03:24:20.751966 8291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:20.753057 8291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:20.754294 8291 hybrid_clock.cc:648] HybridClock initialized: now 1786332260754213 us; error 65 us; skew 500 ppm
I20260810 03:24:20.754736 8291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:20 dist-test-slave-btw8 krb5kdc[8026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:20.758520 8291 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:24:20.760396 8291 webserver.cc:460] Webserver started at http://127.19.246.194:40747/ using document root <none> and password file <none>
I20260810 03:24:20.761185 8291 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:20.761350 8291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:20.761715 8291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:20.765147 8291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "ebac6f81a44c48d18b391571ba9ea0e3"
format_stamp: "Formatted at 2026-08-10 03:24:20 on dist-test-slave-btw8"
server_key: "02832326aedb51fd8dcc1b18ddd02cbe"
server_key_iv: "c3f4a57a1281c9d9f694775976116f2a"
server_key_version: "encryptionkey@0"
I20260810 03:24:20.766081 8291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "ebac6f81a44c48d18b391571ba9ea0e3"
format_stamp: "Formatted at 2026-08-10 03:24:20 on dist-test-slave-btw8"
server_key: "02832326aedb51fd8dcc1b18ddd02cbe"
server_key_iv: "c3f4a57a1281c9d9f694775976116f2a"
server_key_version: "encryptionkey@0"
I20260810 03:24:20.772399 8291 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 03:24:20.776439 8308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:20.777941 8291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:24:20.778151 8291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "ebac6f81a44c48d18b391571ba9ea0e3"
format_stamp: "Formatted at 2026-08-10 03:24:20 on dist-test-slave-btw8"
server_key: "02832326aedb51fd8dcc1b18ddd02cbe"
server_key_iv: "c3f4a57a1281c9d9f694775976116f2a"
server_key_version: "encryptionkey@0"
I20260810 03:24:20.778390 8291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:20.792757 8291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:20.797528 8291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:20.797894 8291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:20.798957 8291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:20.800544 8291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:20.800658 8291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:20.800833 8291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:20.800918 8291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:20.838523 8291 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:42221
I20260810 03:24:20.838587 8435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:42221 every 8 connection(s)
I20260810 03:24:20.840116 8291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:24:20.840691 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 8291
I20260810 03:24:20.840986 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:24:20.841646 20443 external_mini_cluster.cc:1428] Setting key 28a9090c84f17bd7a7e63132f7fa0694
Aug 10 03:24:20 dist-test-slave-btw8 krb5kdc[8026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332260, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:20.868045 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:20.843630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:59005 (local address 127.19.246.254:33257)
0810 03:24:20.844220 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:20.844226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:20.845510 (+ 1284us) server_negotiation.cc:407] Connection header received
0810 03:24:20.847908 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:20.847918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:20.848038 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:20.848362 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:20.850399 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:20.851815 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:20.853639 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:20.854566 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:20.854741 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:20.859452 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:20.859459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:20.859464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:20.859505 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:20.863757 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:20.864820 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:20.864829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:20.864835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:20.865022 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:20.865538 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:20.865546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:20.865552 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:20.866047 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:20.866228 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:20.867149 (+ 921us) server_negotiation.cc:299] Negotiation successful
0810 03:24:20.867486 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":354,"threads_started":1}
I20260810 03:24:20.869148 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:20.844297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:33257 (local address 127.19.246.194:59005)
0810 03:24:20.845312 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:20.845362 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:20.847548 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:20.848736 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:20.848749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:20.849513 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:20.850161 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:20.850184 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:20.852018 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:20.852022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:20.853432 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:20.853448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:20.854862 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:20.854868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:20.855115 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:20.856030 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:20.856058 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:20.859005 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:20.864012 (+ 5007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:20.864022 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:20.864039 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:20.864640 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:20.865157 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:20.865160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:20.865162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:20.865395 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:20.866210 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:20.866217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:20.866612 (+ 395us) client_negotiation.cc:769] Sending connection context
0810 03:24:20.866964 (+ 352us) client_negotiation.cc:240] Negotiation successful
0810 03:24:20.867722 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":425,"threads_started":1}
I20260810 03:24:20.871014 8436 heartbeater.cc:347] Connected to a master server at 127.19.246.254:33257
I20260810 03:24:20.871488 8436 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:20.872426 8436 heartbeater.cc:511] Master 127.19.246.254:33257 requested a full tablet report, sending...
I20260810 03:24:20.874671 8077 ts_manager.cc:194] Registered new tserver with Master: ebac6f81a44c48d18b391571ba9ea0e3 (127.19.246.194:42221)
I20260810 03:24:20.876286 8077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:59005
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:24:20.928321 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:33257
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:45157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:24:21.278733 8282 heartbeater.cc:503] Master 127.19.246.254:33257 was elected leader, sending a full tablet report...
W20260810 03:24:21.312798 8443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:21.313144 8443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:21.313252 8443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:21.321367 8443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:21.321521 8443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:21.321648 8443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:24:21.332556 8443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:45157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:21.334017 8443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:21.335556 8443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:21.351432 8448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:21.351382 8449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:21.351452 8451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:21.352304 8443 server_base.cc:1034] running on GCE node
I20260810 03:24:21.353082 8443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:21.354229 8443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:21.355592 8443 hybrid_clock.cc:648] HybridClock initialized: now 1786332261355416 us; error 75 us; skew 500 ppm
I20260810 03:24:21.356020 8443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:21 dist-test-slave-btw8 krb5kdc[8026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:21.359983 8443 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:24:21.361900 8443 webserver.cc:460] Webserver started at http://127.19.246.195:45505/ using document root <none> and password file <none>
I20260810 03:24:21.362702 8443 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:21.362838 8443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:21.363158 8443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:21.366856 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "a1e65aed9b1d43629711aad2a953daef"
format_stamp: "Formatted at 2026-08-10 03:24:21 on dist-test-slave-btw8"
server_key: "ac183958605877fd01c20116ab2d44d9"
server_key_iv: "e4e28d6c733a99325af699f9443a5386"
server_key_version: "encryptionkey@0"
I20260810 03:24:21.368266 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "a1e65aed9b1d43629711aad2a953daef"
format_stamp: "Formatted at 2026-08-10 03:24:21 on dist-test-slave-btw8"
server_key: "ac183958605877fd01c20116ab2d44d9"
server_key_iv: "e4e28d6c733a99325af699f9443a5386"
server_key_version: "encryptionkey@0"
I20260810 03:24:21.374764 8443 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:24:21.380296 8459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:21.381908 8443 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:24:21.382138 8443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "a1e65aed9b1d43629711aad2a953daef"
format_stamp: "Formatted at 2026-08-10 03:24:21 on dist-test-slave-btw8"
server_key: "ac183958605877fd01c20116ab2d44d9"
server_key_iv: "e4e28d6c733a99325af699f9443a5386"
server_key_version: "encryptionkey@0"
I20260810 03:24:21.382369 8443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:21.419716 8443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:21.425031 8443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:21.425421 8443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:21.426651 8443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:21.428371 8443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:21.428488 8443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:21.428665 8443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:21.428759 8443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:21.471138 8443 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:44395
I20260810 03:24:21.471259 8579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:44395 every 8 connection(s)
I20260810 03:24:21.472863 8443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:24:21.477394 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 8443
I20260810 03:24:21.477643 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:24:21.478219 20443 external_mini_cluster.cc:1428] Setting key 863213724a725dd72be82b3c81076ef3
Aug 10 03:24:21 dist-test-slave-btw8 krb5kdc[8026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:21.498945 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.476075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:56205 (local address 127.19.246.254:33257)
0810 03:24:21.476638 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.476644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.477795 (+ 1151us) server_negotiation.cc:407] Connection header received
0810 03:24:21.480816 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.480822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.480929 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.481073 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:21.483183 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.484188 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.485899 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.486409 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.486602 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.491072 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:21.491079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:21.491083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:21.491126 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:21.494877 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:21.496142 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:21.496161 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:21.496165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:21.496336 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:21.496942 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:21.496947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:21.496949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:21.497327 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:21.497470 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.498405 (+ 935us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.498620 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":294,"threads_started":1}
I20260810 03:24:21.500043 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.476502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:33257 (local address 127.19.246.195:56205)
0810 03:24:21.477576 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:21.477643 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:21.480330 (+ 2687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:21.481394 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:21.481404 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:21.482190 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:21.482952 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.482974 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.484369 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.484379 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.485728 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.485743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.486764 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.486772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.487045 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.487952 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:21.487982 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:21.490731 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:21.495154 (+ 4423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:21.495168 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:21.495193 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:21.495962 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:21.496473 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:21.496477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:21.496479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:21.496768 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:21.497481 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:21.497488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:21.497942 (+ 454us) client_negotiation.cc:769] Sending connection context
0810 03:24:21.498217 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 03:24:21.498707 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":470,"threads_started":1}
I20260810 03:24:21.501978 8580 heartbeater.cc:347] Connected to a master server at 127.19.246.254:33257
I20260810 03:24:21.502700 8580 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:21.503810 8580 heartbeater.cc:511] Master 127.19.246.254:33257 requested a full tablet report, sending...
I20260810 03:24:21.506255 8077 ts_manager.cc:194] Registered new tserver with Master: a1e65aed9b1d43629711aad2a953daef (127.19.246.195:44395)
I20260810 03:24:21.507781 8077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:56205
I20260810 03:24:21.509670 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:24:21.531294 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.514499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.19.246.254:33257)
0810 03:24:21.514888 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.514910 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.515601 (+ 691us) server_negotiation.cc:407] Connection header received
0810 03:24:21.515742 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.515747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.515907 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.516046 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:21.517925 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.518751 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.520311 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.520827 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.520961 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.524322 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:21.524328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:21.524331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:21.524376 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:21.527556 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:21.528668 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:21.528672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:21.528675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:21.528832 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:21.529504 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:21.529508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:21.529511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:21.529922 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:21.530136 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.530630 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.530961 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260810 03:24:21.536334 8077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:24:21.544975 8077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:24:21.582578 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.560115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.19.246.195:44395)
0810 03:24:21.572914 (+ 12799us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.572920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.572949 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:24:21.573030 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.573035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.573217 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.573471 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:21.574389 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.576448 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.579451 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.581119 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.581362 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.581618 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.581948 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.582152 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12652,"thread_start_us":12499,"threads_started":1}
I20260810 03:24:21.582667 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.559519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:44395 (local address 127.0.0.1:43128)
0810 03:24:21.571779 (+ 12260us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:21.571808 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:21.571979 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:21.573479 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:21.573487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:21.573522 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:21.574086 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.574103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.576604 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.576612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.579212 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.579228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.581355 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.581362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.581623 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.581688 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 03:24:21.581847 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 03:24:21.582072 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12080,"mutex_wait_us":41,"thread_start_us":11883,"threads_started":1}
I20260810 03:24:21.583508 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.561775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:39873 (local address 127.0.0.1:50292)
0810 03:24:21.572210 (+ 10435us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:21.572226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:21.572330 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:21.573891 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:21.573897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:21.573928 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:21.574465 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.574481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.576648 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.576653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.578871 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.578887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.581761 (+ 2874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.581791 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.582041 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.582076 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:24:21.582257 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 03:24:21.582423 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10286,"mutex_wait_us":79,"thread_start_us":7127,"threads_started":1}
I20260810 03:24:21.583521 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.562574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.19.246.193:39873)
0810 03:24:21.572701 (+ 10127us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.572712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.572748 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:24:21.572866 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.572880 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.573310 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.573576 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:21.574685 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.576438 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.579721 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.581222 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.581420 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.581579 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.582781 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.583036 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9933,"thread_start_us":9746,"threads_started":1}
I20260810 03:24:21.584000 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.559564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.19.246.194:42221)
0810 03:24:21.574211 (+ 14647us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.574222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.574255 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:24:21.574360 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.574368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.574841 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.575081 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:21.576210 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.577863 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.580605 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.582055 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.582235 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.582408 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.583530 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.583743 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14423,"thread_start_us":14313,"threads_started":1}
I20260810 03:24:21.584329 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.559204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:42221 (local address 127.0.0.1:56346)
0810 03:24:21.569608 (+ 10404us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:21.569675 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:21.569963 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:21.575340 (+ 5377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:21.575378 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:21.575408 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:21.575982 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.575998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.578060 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.578066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.580387 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.580403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.582585 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.582593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.582811 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.582843 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:24:21.582929 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:24:21.583090 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9916,"mutex_wait_us":7,"thread_start_us":12703,"threads_started":1}
I20260810 03:24:21.587597 8511 tablet_service.cc:1467] Processing CreateTablet for tablet 7b07ecc682d141509a1ba5ac8f3720c8 (DEFAULT_TABLE table=test-table [id=b8e18ca6a3e14f15991eedeecd0ccae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:21.587615 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 7b07ecc682d141509a1ba5ac8f3720c8 (DEFAULT_TABLE table=test-table [id=b8e18ca6a3e14f15991eedeecd0ccae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:21.587927 8364 tablet_service.cc:1467] Processing CreateTablet for tablet 7b07ecc682d141509a1ba5ac8f3720c8 (DEFAULT_TABLE table=test-table [id=b8e18ca6a3e14f15991eedeecd0ccae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:21.589154 8511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b07ecc682d141509a1ba5ac8f3720c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:21.589233 8364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b07ecc682d141509a1ba5ac8f3720c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:21.589226 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b07ecc682d141509a1ba5ac8f3720c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:21.608563 8600 tablet_bootstrap.cc:492] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Bootstrap starting.
I20260810 03:24:21.610404 8599 tablet_bootstrap.cc:492] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Bootstrap starting.
I20260810 03:24:21.613042 8601 tablet_bootstrap.cc:492] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Bootstrap starting.
I20260810 03:24:21.614840 8600 tablet_bootstrap.cc:654] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:21.615163 8599 tablet_bootstrap.cc:654] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:21.616480 8600 log.cc:824] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:21.616554 8599 log.cc:824] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:21.617062 8601 tablet_bootstrap.cc:654] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:21.618976 8601 log.cc:824] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:21.623377 8601 tablet_bootstrap.cc:492] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: No bootstrap required, opened a new log
I20260810 03:24:21.623374 8599 tablet_bootstrap.cc:492] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: No bootstrap required, opened a new log
I20260810 03:24:21.623389 8600 tablet_bootstrap.cc:492] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: No bootstrap required, opened a new log
I20260810 03:24:21.623723 8601 ts_tablet_manager.cc:1397] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260810 03:24:21.623835 8599 ts_tablet_manager.cc:1397] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260810 03:24:21.623839 8600 ts_tablet_manager.cc:1397] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.010s
I20260810 03:24:21.631780 8601 raft_consensus.cc:348] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
I20260810 03:24:21.632300 8601 raft_consensus.cc:374] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:21.632405 8601 raft_consensus.cc:729] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebac6f81a44c48d18b391571ba9ea0e3, State: Initialized, Role: FOLLOWER
I20260810 03:24:21.633177 8601 consensus_queue.cc:260] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
I20260810 03:24:21.634500 8436 heartbeater.cc:503] Master 127.19.246.254:33257 was elected leader, sending a full tablet report...
I20260810 03:24:21.635309 8601 ts_tablet_manager.cc:1428] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260810 03:24:21.636482 8600 raft_consensus.cc:348] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
I20260810 03:24:21.636476 8599 raft_consensus.cc:348] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
I20260810 03:24:21.637008 8600 raft_consensus.cc:374] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:21.637009 8599 raft_consensus.cc:374] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:21.637112 8600 raft_consensus.cc:729] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6e96cca73e459b8a5ca698ea4f1611, State: Initialized, Role: FOLLOWER
I20260810 03:24:21.637113 8599 raft_consensus.cc:729] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e65aed9b1d43629711aad2a953daef, State: Initialized, Role: FOLLOWER
I20260810 03:24:21.637883 8600 consensus_queue.cc:260] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
I20260810 03:24:21.638011 8599 consensus_queue.cc:260] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
I20260810 03:24:21.640334 8600 ts_tablet_manager.cc:1428] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260810 03:24:21.640599 8599 ts_tablet_manager.cc:1428] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260810 03:24:21.641150 8580 heartbeater.cc:503] Master 127.19.246.254:33257 was elected leader, sending a full tablet report...
W20260810 03:24:21.725018 8581 tablet.cc:2369] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:21.739777 8607 raft_consensus.cc:482] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:24:21.740145 8607 raft_consensus.cc:504] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
W20260810 03:24:21.742022 8285 tablet.cc:2369] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:21.742275 8607 leader_election.cc:288] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebac6f81a44c48d18b391571ba9ea0e3 (127.19.246.194:42221), a1e65aed9b1d43629711aad2a953daef (127.19.246.195:44395)
I20260810 03:24:21.751861 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.742939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:47353 (local address 127.19.246.194:42221)
0810 03:24:21.743178 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.743188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.744140 (+ 952us) server_negotiation.cc:407] Connection header received
0810 03:24:21.744355 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.744361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.744435 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.744538 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:21.745246 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.746286 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.749037 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.750511 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.750689 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.750773 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.751414 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.751566 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 03:24:21.751861 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.743091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:55455 (local address 127.19.246.195:44395)
0810 03:24:21.743310 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.743316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.743824 (+ 508us) server_negotiation.cc:407] Connection header received
0810 03:24:21.744079 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.744084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.744143 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.744247 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:21.744903 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.746040 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.749037 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.750481 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.750700 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.750773 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.751414 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.751566 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:24:21.752811 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.743169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:42221 (local address 127.19.246.193:47353)
0810 03:24:21.744030 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:21.744045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:21.744212 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:21.744560 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:21.744565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:21.744590 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:21.745064 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.745077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.746413 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.746419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.748761 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.748779 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.750878 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.750886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.751117 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.751198 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 03:24:21.752100 (+ 902us) client_negotiation.cc:240] Negotiation successful
0810 03:24:21.752306 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":141,"thread_start_us":527,"threads_started":1}
I20260810 03:24:21.752853 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.742612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:44395 (local address 127.19.246.193:55455)
0810 03:24:21.743683 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:21.743705 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:21.743922 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:21.744267 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:21.744271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:21.744298 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:21.744688 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.744708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.746187 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.746190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.748762 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:21.748779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.750879 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.750886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:21.751117 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.751203 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 03:24:21.752100 (+ 897us) client_negotiation.cc:240] Negotiation successful
0810 03:24:21.752298 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":108,"spinlock_wait_cycles":18176,"thread_start_us":862,"threads_started":1}
I20260810 03:24:21.754583 8532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b07ecc682d141509a1ba5ac8f3720c8" candidate_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e65aed9b1d43629711aad2a953daef" is_pre_election: true
I20260810 03:24:21.754776 8387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b07ecc682d141509a1ba5ac8f3720c8" candidate_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" is_pre_election: true
I20260810 03:24:21.755281 8387 raft_consensus.cc:2454] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee6e96cca73e459b8a5ca698ea4f1611 in term 0.
I20260810 03:24:21.755261 8532 raft_consensus.cc:2454] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee6e96cca73e459b8a5ca698ea4f1611 in term 0.
I20260810 03:24:21.756140 8160 leader_election.cc:302] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a1e65aed9b1d43629711aad2a953daef, ebac6f81a44c48d18b391571ba9ea0e3, ee6e96cca73e459b8a5ca698ea4f1611; no voters:
I20260810 03:24:21.756781 8607 raft_consensus.cc:2781] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:24:21.757012 8607 raft_consensus.cc:482] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:24:21.757150 8607 raft_consensus.cc:3037] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:21.759680 8607 raft_consensus.cc:504] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
I20260810 03:24:21.760354 8607 leader_election.cc:288] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [CANDIDATE]: Term 1 election: Requested vote from peers ebac6f81a44c48d18b391571ba9ea0e3 (127.19.246.194:42221), a1e65aed9b1d43629711aad2a953daef (127.19.246.195:44395)
I20260810 03:24:21.761111 8532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b07ecc682d141509a1ba5ac8f3720c8" candidate_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e65aed9b1d43629711aad2a953daef"
I20260810 03:24:21.761080 8387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b07ecc682d141509a1ba5ac8f3720c8" candidate_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebac6f81a44c48d18b391571ba9ea0e3"
I20260810 03:24:21.761358 8532 raft_consensus.cc:3037] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:21.761358 8387 raft_consensus.cc:3037] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:21.765060 8532 raft_consensus.cc:2454] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6e96cca73e459b8a5ca698ea4f1611 in term 1.
I20260810 03:24:21.765050 8387 raft_consensus.cc:2454] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee6e96cca73e459b8a5ca698ea4f1611 in term 1.
I20260810 03:24:21.765755 8163 leader_election.cc:302] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a1e65aed9b1d43629711aad2a953daef, ebac6f81a44c48d18b391571ba9ea0e3, ee6e96cca73e459b8a5ca698ea4f1611; no voters:
I20260810 03:24:21.766222 8607 raft_consensus.cc:2781] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:21.766881 8607 raft_consensus.cc:686] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 1 LEADER]: Becoming Leader. State: Replica: ee6e96cca73e459b8a5ca698ea4f1611, State: Running, Role: LEADER
I20260810 03:24:21.767604 8607 consensus_queue.cc:237] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } }
I20260810 03:24:21.772665 8076 catalog_manager.cc:5495] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee6e96cca73e459b8a5ca698ea4f1611 (127.19.246.193). New cstate: current_term: 1 leader_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee6e96cca73e459b8a5ca698ea4f1611" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 39873 } health_report { overall_health: HEALTHY } } }
I20260810 03:24:21.828902 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.822517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.19.246.193:39873)
0810 03:24:21.822759 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.822764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.822786 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:24:21.823001 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.823005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.823101 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.823239 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:24:21.823831 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.824602 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.826182 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.826630 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.826744 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.827433 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:24:21.828057 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:24:21.828202 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.828509 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.828572 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:24:21.839152 8532 raft_consensus.cc:1261] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 1 FOLLOWER]: Refusing update from remote peer ee6e96cca73e459b8a5ca698ea4f1611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:21.839154 8387 raft_consensus.cc:1261] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 1 FOLLOWER]: Refusing update from remote peer ee6e96cca73e459b8a5ca698ea4f1611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:21.840552 8612 consensus_queue.cc:1035] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e65aed9b1d43629711aad2a953daef" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 44395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:21.841033 8607 consensus_queue.cc:1035] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebac6f81a44c48d18b391571ba9ea0e3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 42221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:24:21.842986 8438 tablet.cc:2369] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:21.863217 8615 mvcc.cc:205] Tried to move back new op lower bound from 7316816944478973952 to 7316816944204746752. Current Snapshot: MvccSnapshot[applied={T|T < 7316816944478973952}]
I20260810 03:24:21.880738 8076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:21.881140 8076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:21.886078 8076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff3ef6ba9b64589aaccafe273f83df3: Sending DeleteTablet for 3 replicas of tablet 7b07ecc682d141509a1ba5ac8f3720c8
I20260810 03:24:21.887765 8511 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b07ecc682d141509a1ba5ac8f3720c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:21 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:43128
I20260810 03:24:21.888002 8364 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b07ecc682d141509a1ba5ac8f3720c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:21 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:56346
I20260810 03:24:21.888610 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b07ecc682d141509a1ba5ac8f3720c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:21 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:50292
I20260810 03:24:21.889711 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 8143
I20260810 03:24:21.903570 8625 tablet_replica.cc:330] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: stopping tablet replica
I20260810 03:24:21.904268 8625 raft_consensus.cc:2229] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:21.904934 8625 raft_consensus.cc:2258] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:21.905339 8624 tablet_replica.cc:330] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: stopping tablet replica
I20260810 03:24:21.907676 8624 raft_consensus.cc:2229] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:21.908214 8626 tablet_replica.cc:330] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: stopping tablet replica
I20260810 03:24:21.908835 8626 raft_consensus.cc:2229] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:24:21.909053 8624 raft_consensus.cc:2258] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:21.909709 8626 raft_consensus.cc:2258] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:21.912711 8626 ts_tablet_manager.cc:1905] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:21.915300 8624 ts_tablet_manager.cc:1905] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 03:24:21 dist-test-slave-btw8 krb5kdc[8026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:24:21.922534 8626 ts_tablet_manager.cc:1918] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:21.922883 8626 log.cc:1196] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/7b07ecc682d141509a1ba5ac8f3720c8
I20260810 03:24:21.923413 8624 ts_tablet_manager.cc:1918] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:21.923599 8624 log.cc:1196] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/7b07ecc682d141509a1ba5ac8f3720c8
I20260810 03:24:21.923882 8626 ts_tablet_manager.cc:1939] T 7b07ecc682d141509a1ba5ac8f3720c8 P ee6e96cca73e459b8a5ca698ea4f1611: Deleting consensus metadata
I20260810 03:24:21.924158 8624 ts_tablet_manager.cc:1939] T 7b07ecc682d141509a1ba5ac8f3720c8 P a1e65aed9b1d43629711aad2a953daef: Deleting consensus metadata
I20260810 03:24:21.924465 8625 ts_tablet_manager.cc:1905] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:21.926177 8057 catalog_manager.cc:4841] TS a1e65aed9b1d43629711aad2a953daef (127.19.246.195:44395): tablet 7b07ecc682d141509a1ba5ac8f3720c8 (table test-table [id=b8e18ca6a3e14f15991eedeecd0ccae2]) successfully deleted
I20260810 03:24:21.926553 8058 catalog_manager.cc:4841] TS ee6e96cca73e459b8a5ca698ea4f1611 (127.19.246.193:39873): tablet 7b07ecc682d141509a1ba5ac8f3720c8 (table test-table [id=b8e18ca6a3e14f15991eedeecd0ccae2]) successfully deleted
I20260810 03:24:21.930665 8625 ts_tablet_manager.cc:1918] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:21.930892 8625 log.cc:1196] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/7b07ecc682d141509a1ba5ac8f3720c8
I20260810 03:24:21.931519 8625 ts_tablet_manager.cc:1939] T 7b07ecc682d141509a1ba5ac8f3720c8 P ebac6f81a44c48d18b391571ba9ea0e3: Deleting consensus metadata
I20260810 03:24:21.933543 8061 catalog_manager.cc:4841] TS ebac6f81a44c48d18b391571ba9ea0e3 (127.19.246.194:42221): tablet 7b07ecc682d141509a1ba5ac8f3720c8 (table test-table [id=b8e18ca6a3e14f15991eedeecd0ccae2]) successfully deleted
I20260810 03:24:21.940131 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:21.893709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.19.246.193:39873)
0810 03:24:21.894005 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:21.894016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:21.904565 (+ 10549us) server_negotiation.cc:407] Connection header received
0810 03:24:21.909648 (+ 5083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:21.909655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:21.909786 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:21.909879 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:21.912074 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.913372 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.915477 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:21.916322 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:21.916432 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:21.931049 (+ 14617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:21.931074 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:21.931097 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:21.931174 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:21.936020 (+ 4846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:21.937297 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:21.937306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:21.937308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:21.937586 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:21.938099 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:21.938105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:21.938108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:21.938577 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:21.938952 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:21.939320 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 03:24:21.939700 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 03:24:21.940968 8277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:22.048663 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 8143
I20260810 03:24:22.073701 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 8291
Aug 10 03:24:22 dist-test-slave-btw8 krb5kdc[8026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:24:22.092777 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:22.074772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.19.246.194:42221)
0810 03:24:22.074980 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:22.074985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:22.075007 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:24:22.075240 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:22.075245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:22.075330 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:22.075510 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:22.077012 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:22.077866 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:22.079450 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:22.079971 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:22.080088 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:22.085838 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:22.085857 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:22.085873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:22.085923 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:22.089330 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:22.090250 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:22.090258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:22.090261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:22.090409 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:22.090895 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:22.090899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:22.090902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:22.091342 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:22.091777 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:22.092129 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 03:24:22.092404 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:24:22.093554 8431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:22.193538 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 8291
I20260810 03:24:22.214749 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 8443
Aug 10 03:24:22 dist-test-slave-btw8 krb5kdc[8026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:24:22.233299 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:22.215759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.19.246.195:44395)
0810 03:24:22.215959 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:22.215963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:22.215984 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:24:22.216141 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:22.216147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:22.216248 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:22.216344 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:22.217953 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:22.218778 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:22.220349 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:22.220750 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:22.220857 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:22.226332 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:22.226352 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:22.226370 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:22.226413 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:22.229652 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:22.230580 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:22.230588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:22.230590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:22.230767 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:22.231297 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:22.231301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:22.231303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:22.232060 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:22.232320 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:22.232695 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 03:24:22.232930 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:24:22.234211 8574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:22.329551 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 8443
I20260810 03:24:22.350981 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 8040
I20260810 03:24:22.352213 8109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:22.470311 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 8040
2026-08-10T03:24:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5151 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T03:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:22Z Disabled control of system clock
I20260810 03:24:22.575143 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:37747
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:37747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:24:22.946151 8636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:22.946587 8636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:22.946753 8636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:22.954571 8636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:22.954784 8636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:22.954875 8636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:22.954941 8636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:22.954986 8636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:22.965696 8636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:37747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:22.967191 8636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:22.969638 8636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:22.984994 8641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:22.984889 8644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:22.984964 8642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:22.986588 8636 server_base.cc:1034] running on GCE node
I20260810 03:24:22.987905 8636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:22.989789 8636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:22.991114 8636 hybrid_clock.cc:648] HybridClock initialized: now 1786332262991023 us; error 75 us; skew 500 ppm
I20260810 03:24:22.991672 8636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:22.994840 8636 webserver.cc:460] Webserver started at http://127.19.246.254:36119/ using document root <none> and password file <none>
I20260810 03:24:22.995827 8636 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:22.995997 8636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:22.996400 8636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:23.000101 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "98ca21877e2f40a2be087e70fc5c2a40"
format_stamp: "Formatted at 2026-08-10 03:24:22 on dist-test-slave-btw8"
server_key: "dd94dbf0db904cc81bc1d4a652293c76"
server_key_iv: "70c6b0a0acdc909c0ecbc236b10a773f"
server_key_version: "encryptionkey@0"
I20260810 03:24:23.001179 8636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "98ca21877e2f40a2be087e70fc5c2a40"
format_stamp: "Formatted at 2026-08-10 03:24:22 on dist-test-slave-btw8"
server_key: "dd94dbf0db904cc81bc1d4a652293c76"
server_key_iv: "70c6b0a0acdc909c0ecbc236b10a773f"
server_key_version: "encryptionkey@0"
I20260810 03:24:23.008019 8636 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:24:23.012944 8651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:23.014884 8636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:24:23.015153 8636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "98ca21877e2f40a2be087e70fc5c2a40"
format_stamp: "Formatted at 2026-08-10 03:24:22 on dist-test-slave-btw8"
server_key: "dd94dbf0db904cc81bc1d4a652293c76"
server_key_iv: "70c6b0a0acdc909c0ecbc236b10a773f"
server_key_version: "encryptionkey@0"
I20260810 03:24:23.015471 8636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:23.043148 8636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:23.044252 8636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:23.044615 8636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:23.067039 8636 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:37747
I20260810 03:24:23.067059 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:37747 every 8 connection(s)
I20260810 03:24:23.068645 8636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:23.071846 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 8636
I20260810 03:24:23.072031 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:23.072508 20443 external_mini_cluster.cc:1428] Setting key f7bef1daf1ba66e231ebfe8c7803165c
I20260810 03:24:23.073490 8708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:23.090533 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:23.074505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.19.246.254:37747)
0810 03:24:23.075683 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:23.075701 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:23.075767 (+ 66us) server_negotiation.cc:407] Connection header received
0810 03:24:23.077628 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:23.077662 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:23.078422 (+ 760us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:23.078943 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:23.079605 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:23.081996 (+ 2391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:23.084070 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:23.085267 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:23.085495 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:23.087009 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:23.087020 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:23.087048 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:23.087958 (+ 910us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:23.088074 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:23.088106 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:23.088272 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:23.088648 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 03:24:23.089055 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":358,"threads_started":1}
I20260810 03:24:23.091107 8708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40: Bootstrap starting.
I20260810 03:24:23.095098 8708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:23.096302 8708 log.cc:824] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:23.099730 8708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40: No bootstrap required, opened a new log
I20260810 03:24:23.108177 8708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ca21877e2f40a2be087e70fc5c2a40" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 37747 } }
I20260810 03:24:23.108667 8708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:23.108788 8708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ca21877e2f40a2be087e70fc5c2a40, State: Initialized, Role: FOLLOWER
I20260810 03:24:23.109444 8708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ca21877e2f40a2be087e70fc5c2a40" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 37747 } }
I20260810 03:24:23.109723 8708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:23.109886 8708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:23.110078 8708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:23.112795 8708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ca21877e2f40a2be087e70fc5c2a40" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 37747 } }
I20260810 03:24:23.113462 8708 leader_election.cc:302] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ca21877e2f40a2be087e70fc5c2a40; no voters:
I20260810 03:24:23.114208 8708 leader_election.cc:288] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:23.114420 8715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:23.115500 8715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [term 1 LEADER]: Becoming Leader. State: Replica: 98ca21877e2f40a2be087e70fc5c2a40, State: Running, Role: LEADER
I20260810 03:24:23.116236 8715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ca21877e2f40a2be087e70fc5c2a40" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 37747 } }
I20260810 03:24:23.116807 8708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:23.119717 8717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ca21877e2f40a2be087e70fc5c2a40. Latest consensus state: current_term: 1 leader_uuid: "98ca21877e2f40a2be087e70fc5c2a40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ca21877e2f40a2be087e70fc5c2a40" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 37747 } } }
I20260810 03:24:23.120195 8717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:23.125552 8716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98ca21877e2f40a2be087e70fc5c2a40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ca21877e2f40a2be087e70fc5c2a40" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 37747 } } }
I20260810 03:24:23.125947 8716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:23.127012 8722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:23.133627 8722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:23.145802 8722 catalog_manager.cc:1347] Generated new cluster ID: 1827089d460a4aa493aaecbf698ff224
I20260810 03:24:23.145970 8722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:23.164368 8722 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:23.165892 8722 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:23.178268 8722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40: Generated new TSK 0
I20260810 03:24:23.179260 8722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:23.186820 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:37747
--builtin_ntp_servers=127.19.246.212:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:24:23.570463 8735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:23.570804 8735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:23.570962 8735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:23.578768 8735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:23.578943 8735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:23.579140 8735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:24:23.589092 8735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:37747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:23.590588 8735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:23.592689 8735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:23.607087 8740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:23.607043 8741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:23.607141 8743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:23.608276 8735 server_base.cc:1034] running on GCE node
I20260810 03:24:23.608992 8735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:23.610162 8735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:23.611492 8735 hybrid_clock.cc:648] HybridClock initialized: now 1786332263611413 us; error 59 us; skew 500 ppm
I20260810 03:24:23.611868 8735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:23.614688 8735 webserver.cc:460] Webserver started at http://127.19.246.193:39127/ using document root <none> and password file <none>
I20260810 03:24:23.615622 8735 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:23.615753 8735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:23.616119 8735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:23.619446 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "d88de99b95a44a888c4b8bf2f79e2ce6"
format_stamp: "Formatted at 2026-08-10 03:24:23 on dist-test-slave-btw8"
server_key: "cd92627a18929de757014238bc720305"
server_key_iv: "f9e1697464d83f4df7e8d3bd63d3a13d"
server_key_version: "encryptionkey@0"
I20260810 03:24:23.620640 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "d88de99b95a44a888c4b8bf2f79e2ce6"
format_stamp: "Formatted at 2026-08-10 03:24:23 on dist-test-slave-btw8"
server_key: "cd92627a18929de757014238bc720305"
server_key_iv: "f9e1697464d83f4df7e8d3bd63d3a13d"
server_key_version: "encryptionkey@0"
I20260810 03:24:23.626765 8735 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:24:23.630801 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:23.632539 8735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:24:23.632762 8735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "d88de99b95a44a888c4b8bf2f79e2ce6"
format_stamp: "Formatted at 2026-08-10 03:24:23 on dist-test-slave-btw8"
server_key: "cd92627a18929de757014238bc720305"
server_key_iv: "f9e1697464d83f4df7e8d3bd63d3a13d"
server_key_version: "encryptionkey@0"
I20260810 03:24:23.633026 8735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:23.657191 8735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:23.658176 8735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:23.658545 8735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:23.659804 8735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:23.661480 8735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:23.661623 8735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:23.661777 8735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:23.661878 8735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:23.699753 8735 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:46803
I20260810 03:24:23.699855 8870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:46803 every 8 connection(s)
I20260810 03:24:23.701188 8735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:24:23.705506 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 8735
I20260810 03:24:23.705727 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:24:23.706385 20443 external_mini_cluster.cc:1428] Setting key e7b8485032b8b7cd7d2b68129658292f
I20260810 03:24:23.709355 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:37747
--builtin_ntp_servers=127.19.246.212:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:24:23.716141 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:23.704465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:59681 (local address 127.19.246.254:37747)
0810 03:24:23.704997 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:23.705003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:23.706911 (+ 1908us) server_negotiation.cc:407] Connection header received
0810 03:24:23.708894 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:23.708900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:23.708987 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:23.709131 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:23.710394 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:23.711180 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:23.712936 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:23.713368 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:23.713514 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:23.715180 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:23.715185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:23.715188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:23.715210 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:23.715252 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:23.715265 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:23.715385 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:23.715636 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 03:24:23.715747 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
I20260810 03:24:23.717363 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:23.705386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:37747 (local address 127.19.246.193:59681)
0810 03:24:23.706730 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:23.706772 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:23.708601 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:23.709397 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:23.709405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:23.709573 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:23.710209 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:23.710227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:23.711438 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:23.711444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:23.712774 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:23.712787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:23.713629 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:23.713637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:23.713893 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:23.714781 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:23.714784 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:23.714830 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:23.714928 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:23.715393 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:23.715406 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:23.715417 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 03:24:23.715633 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 03:24:23.716078 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":375,"threads_started":1}
I20260810 03:24:23.719214 8871 heartbeater.cc:347] Connected to a master server at 127.19.246.254:37747
I20260810 03:24:23.719583 8871 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:23.720506 8871 heartbeater.cc:511] Master 127.19.246.254:37747 requested a full tablet report, sending...
I20260810 03:24:23.722993 8671 ts_manager.cc:194] Registered new tserver with Master: d88de99b95a44a888c4b8bf2f79e2ce6 (127.19.246.193:46803)
I20260810 03:24:23.724923 8671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.193:59681
W20260810 03:24:24.101752 8875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:24.102099 8875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:24.102252 8875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:24.110311 8875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:24.110457 8875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:24.110641 8875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:24:24.125677 8875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:37747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:24.127303 8875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:24.129326 8875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:24.145083 8880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:24.145063 8881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:24.145884 8883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:24.146783 8875 server_base.cc:1034] running on GCE node
I20260810 03:24:24.147661 8875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:24.148764 8875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:24.150032 8875 hybrid_clock.cc:648] HybridClock initialized: now 1786332264149952 us; error 77 us; skew 500 ppm
I20260810 03:24:24.150420 8875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:24.159046 8875 webserver.cc:460] Webserver started at http://127.19.246.194:39873/ using document root <none> and password file <none>
I20260810 03:24:24.159993 8875 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:24.160166 8875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:24.160539 8875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:24.164477 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2"
format_stamp: "Formatted at 2026-08-10 03:24:24 on dist-test-slave-btw8"
server_key: "b6668461e4c54a98282ece23807a294d"
server_key_iv: "38e49b0852b97785b357e0186b4fe55d"
server_key_version: "encryptionkey@0"
I20260810 03:24:24.165429 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2"
format_stamp: "Formatted at 2026-08-10 03:24:24 on dist-test-slave-btw8"
server_key: "b6668461e4c54a98282ece23807a294d"
server_key_iv: "38e49b0852b97785b357e0186b4fe55d"
server_key_version: "encryptionkey@0"
I20260810 03:24:24.172076 8875 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 03:24:24.176254 8889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:24.177815 8875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:24:24.178059 8875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2"
format_stamp: "Formatted at 2026-08-10 03:24:24 on dist-test-slave-btw8"
server_key: "b6668461e4c54a98282ece23807a294d"
server_key_iv: "38e49b0852b97785b357e0186b4fe55d"
server_key_version: "encryptionkey@0"
I20260810 03:24:24.178381 8875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:24.205865 8875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:24.207008 8875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:24.207489 8875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:24.208827 8875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:24.210614 8875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:24.210764 8875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:24.210918 8875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:24.211022 8875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:24.251040 8875 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:45743
I20260810 03:24:24.251106 9016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:45743 every 8 connection(s)
I20260810 03:24:24.252553 8875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:24:24.259492 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 8875
I20260810 03:24:24.259725 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:24:24.260267 20443 external_mini_cluster.cc:1428] Setting key 9c4cae4bceef60b20204e409aa500367
I20260810 03:24:24.264400 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:37747
--builtin_ntp_servers=127.19.246.212:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:24:24.268730 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.255536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:53709 (local address 127.19.246.254:37747)
0810 03:24:24.256154 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:24.256159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:24.257199 (+ 1040us) server_negotiation.cc:407] Connection header received
0810 03:24:24.259306 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:24.259312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:24.259450 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:24.259581 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:24.260949 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.262546 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.264783 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.265573 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.265767 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.267542 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:24.267551 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:24.267558 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:24.267588 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:24.267646 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:24.267666 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:24.267770 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:24.268031 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 03:24:24.268245 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":347,"threads_started":1}
I20260810 03:24:24.269658 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.256014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:37747 (local address 127.19.246.194:53709)
0810 03:24:24.256996 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:24.257035 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:24.258961 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:24.259913 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:24.259927 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:24.260041 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:24.260750 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.260769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.262887 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.262893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.264490 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.264511 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.265915 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.265922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.266200 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.267064 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:24.267067 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:24.267129 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:24.267234 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:24.267779 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:24.267788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:24.267795 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:24:24.267980 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 03:24:24.268360 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":304,"threads_started":1}
I20260810 03:24:24.271920 9017 heartbeater.cc:347] Connected to a master server at 127.19.246.254:37747
I20260810 03:24:24.272337 9017 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:24.273200 9017 heartbeater.cc:511] Master 127.19.246.254:37747 requested a full tablet report, sending...
I20260810 03:24:24.275019 8671 ts_manager.cc:194] Registered new tserver with Master: 1794ed8d6fba4ce3b3ba127fd4d351e2 (127.19.246.194:45743)
I20260810 03:24:24.276039 8671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.194:53709
W20260810 03:24:24.639096 9021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:24.639508 9021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:24.639671 9021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:24.647321 9021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:24.647534 9021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:24.647730 9021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:24:24.657591 9021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:37747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:24.659066 9021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:24.661188 9021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:24.675810 9027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:24.675905 9029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:24.676380 9026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:24.677433 9021 server_base.cc:1034] running on GCE node
I20260810 03:24:24.678170 9021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:24.679256 9021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:24.680559 9021 hybrid_clock.cc:648] HybridClock initialized: now 1786332264680518 us; error 47 us; skew 500 ppm
I20260810 03:24:24.680924 9021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:24.683872 9021 webserver.cc:460] Webserver started at http://127.19.246.195:45037/ using document root <none> and password file <none>
I20260810 03:24:24.684674 9021 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:24.684840 9021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:24.685194 9021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:24.688512 9021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "2e156ed81f844facaf1cef6da27bcab0"
format_stamp: "Formatted at 2026-08-10 03:24:24 on dist-test-slave-btw8"
server_key: "489f8cde2e6c6d1296f9f2835255d1ed"
server_key_iv: "21e4d0795580feaf2a2418918a914c15"
server_key_version: "encryptionkey@0"
I20260810 03:24:24.689438 9021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "2e156ed81f844facaf1cef6da27bcab0"
format_stamp: "Formatted at 2026-08-10 03:24:24 on dist-test-slave-btw8"
server_key: "489f8cde2e6c6d1296f9f2835255d1ed"
server_key_iv: "21e4d0795580feaf2a2418918a914c15"
server_key_version: "encryptionkey@0"
I20260810 03:24:24.695497 9021 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260810 03:24:24.699901 9035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:24.701828 9021 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:24:24.702046 9021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "2e156ed81f844facaf1cef6da27bcab0"
format_stamp: "Formatted at 2026-08-10 03:24:24 on dist-test-slave-btw8"
server_key: "489f8cde2e6c6d1296f9f2835255d1ed"
server_key_iv: "21e4d0795580feaf2a2418918a914c15"
server_key_version: "encryptionkey@0"
I20260810 03:24:24.702332 9021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:24.724555 9021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:24.725543 9021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:24.725869 9021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:24.727118 9021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:24.728765 8871 heartbeater.cc:503] Master 127.19.246.254:37747 was elected leader, sending a full tablet report...
I20260810 03:24:24.729290 9021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:24.729429 9021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:24.729573 9021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:24.729682 9021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:24.767906 9021 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:41575
I20260810 03:24:24.768024 9159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:41575 every 8 connection(s)
I20260810 03:24:24.769347 9021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:24:24.774255 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 9021
I20260810 03:24:24.774461 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:24:24.775038 20443 external_mini_cluster.cc:1428] Setting key 62b5a6f404464738bcd3d8a9787ffbc7
I20260810 03:24:24.783769 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.772264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:45889 (local address 127.19.246.254:37747)
0810 03:24:24.772799 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:24.772804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:24.774134 (+ 1330us) server_negotiation.cc:407] Connection header received
0810 03:24:24.776539 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:24.776545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:24.776647 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:24.776771 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:24.778216 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.778994 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.780708 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.781102 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.781241 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.782714 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:24.782719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:24.782722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:24.782743 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:24.782790 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:24.782802 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:24.782910 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:24.783217 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 03:24:24.783325 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":289,"threads_started":1}
I20260810 03:24:24.784842 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.772893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:37747 (local address 127.19.246.195:45889)
0810 03:24:24.773905 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:24.773958 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:24.776217 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:24.777035 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:24.777044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:24.777137 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:24.778043 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.778065 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.779166 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.779172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.780543 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.780556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.781368 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.781377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.781607 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.782319 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:24.782322 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:24.782372 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:24.782465 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:24.782927 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:24.782940 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:24.782952 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:24:24.783190 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 03:24:24.783659 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":277,"threads_started":1}
I20260810 03:24:24.786623 9160 heartbeater.cc:347] Connected to a master server at 127.19.246.254:37747
I20260810 03:24:24.787030 9160 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:24.787984 9160 heartbeater.cc:511] Master 127.19.246.254:37747 requested a full tablet report, sending...
I20260810 03:24:24.790079 8671 ts_manager.cc:194] Registered new tserver with Master: 2e156ed81f844facaf1cef6da27bcab0 (127.19.246.195:41575)
I20260810 03:24:24.791307 8671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.195:45889
I20260810 03:24:24.791628 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:24:24.803630 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.796811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.19.246.254:37747)
0810 03:24:24.797015 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:24.797020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:24.797563 (+ 543us) server_negotiation.cc:407] Connection header received
0810 03:24:24.797701 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:24.797708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:24.797777 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:24.797872 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:24.798457 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.799218 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.800785 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.801234 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.801367 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.802825 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:24.802830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:24.802833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:24.802854 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:24.802891 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:24.802904 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:24.803107 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:24.803282 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 03:24:24.803345 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:24:24.808370 8671 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:24:24.815743 8671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:24:24.853971 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.831095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46803 (local address 127.0.0.1:56254)
0810 03:24:24.841413 (+ 10318us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:24.841434 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:24.841540 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:24.846222 (+ 4682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:24.846228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:24.846264 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:24.846716 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.846730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.848847 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.848853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.851034 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.851048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.852686 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.852693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.852973 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.853133 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 03:24:24.853292 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 03:24:24.853526 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10200,"mutex_wait_us":7,"thread_start_us":10519,"threads_started":1}
I20260810 03:24:24.854455 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.831475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:41575 (local address 127.0.0.1:39528)
0810 03:24:24.842376 (+ 10901us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:24.842395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:24.842506 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:24.845221 (+ 2715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:24.845234 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:24.845290 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:24.845955 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.845974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.848397 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.848405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.850992 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.851013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.853493 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.853499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.853674 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.853705 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:24:24.853798 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 03:24:24.853941 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10765,"thread_start_us":9650,"threads_started":1}
I20260810 03:24:24.855367 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.832376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.19.246.195:41575)
0810 03:24:24.843917 (+ 11541us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:24.843925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:24.843959 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:24:24.844047 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:24.844053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:24.844302 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:24.846390 (+ 2088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:24.846756 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.848196 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.851291 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.852837 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.853105 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.853346 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:24.854129 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 03:24:24.854816 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11382,"thread_start_us":11218,"threads_started":1}
I20260810 03:24:24.855399 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.831856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.19.246.193:46803)
0810 03:24:24.845066 (+ 13210us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:24.845075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:24.845114 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:24:24.845329 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:24.845339 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:24.845627 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:24.845875 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:24.847049 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.848682 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.851322 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.852441 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.852685 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.852957 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:24.853397 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 03:24:24.854416 (+ 1019us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13016,"thread_start_us":12799,"threads_started":1}
I20260810 03:24:24.856438 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.831009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:45743 (local address 127.0.0.1:57820)
0810 03:24:24.840280 (+ 9271us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:24.840358 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:24.840670 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:24.847678 (+ 7008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:24.847683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:24.847712 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:24.848120 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.848132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.850700 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.850706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.852929 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:24.852948 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.855649 (+ 2701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.855659 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:24.855860 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.855900 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:24:24.855984 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 03:24:24.856134 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8766,"thread_start_us":11089,"threads_started":1}
I20260810 03:24:24.857824 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:24.831273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.19.246.194:45743)
0810 03:24:24.846918 (+ 15645us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:24.846926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:24.846953 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:24:24.847053 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:24.847060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:24.847414 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:24.847669 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:24.848319 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.850525 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.853199 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:24.855274 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:24.856910 (+ 1636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:24.857166 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:24.857242 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 03:24:24.857445 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15468,"thread_start_us":15341,"threads_started":1}
I20260810 03:24:24.859314 8803 tablet_service.cc:1467] Processing CreateTablet for tablet a1ace1a32c5644d98e363f9519e52a3d (DEFAULT_TABLE table=test-table [id=237459380854463bbf0609f861a55113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:24.860327 9090 tablet_service.cc:1467] Processing CreateTablet for tablet a1ace1a32c5644d98e363f9519e52a3d (DEFAULT_TABLE table=test-table [id=237459380854463bbf0609f861a55113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:24.861012 8803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ace1a32c5644d98e363f9519e52a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:24.861404 8944 tablet_service.cc:1467] Processing CreateTablet for tablet a1ace1a32c5644d98e363f9519e52a3d (DEFAULT_TABLE table=test-table [id=237459380854463bbf0609f861a55113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:24.861788 9090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ace1a32c5644d98e363f9519e52a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:24.862337 8944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ace1a32c5644d98e363f9519e52a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:24.876181 9182 tablet_bootstrap.cc:492] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Bootstrap starting.
I20260810 03:24:24.878561 9184 tablet_bootstrap.cc:492] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Bootstrap starting.
I20260810 03:24:24.880527 9182 tablet_bootstrap.cc:654] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:24.882123 9182 log.cc:824] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:24.883093 9183 tablet_bootstrap.cc:492] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Bootstrap starting.
I20260810 03:24:24.883937 9184 tablet_bootstrap.cc:654] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:24.885010 9184 log.cc:824] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:24.888195 9183 tablet_bootstrap.cc:654] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:24.888434 9182 tablet_bootstrap.cc:492] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: No bootstrap required, opened a new log
I20260810 03:24:24.888592 9184 tablet_bootstrap.cc:492] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: No bootstrap required, opened a new log
I20260810 03:24:24.888793 9182 ts_tablet_manager.cc:1397] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260810 03:24:24.888886 9184 ts_tablet_manager.cc:1397] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260810 03:24:24.889602 9183 log.cc:824] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:24.897119 9183 tablet_bootstrap.cc:492] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: No bootstrap required, opened a new log
I20260810 03:24:24.897487 9183 ts_tablet_manager.cc:1397] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260810 03:24:24.897989 9182 raft_consensus.cc:348] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:24.898422 9182 raft_consensus.cc:374] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:24.898535 9182 raft_consensus.cc:729] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1794ed8d6fba4ce3b3ba127fd4d351e2, State: Initialized, Role: FOLLOWER
I20260810 03:24:24.899250 9182 consensus_queue.cc:260] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:24.900696 9017 heartbeater.cc:503] Master 127.19.246.254:37747 was elected leader, sending a full tablet report...
I20260810 03:24:24.901311 9182 ts_tablet_manager.cc:1428] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Time spent starting tablet: real 0.012s user 0.003s sys 0.010s
I20260810 03:24:24.902535 9184 raft_consensus.cc:348] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:24.903611 9184 raft_consensus.cc:374] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:24.903714 9184 raft_consensus.cc:729] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88de99b95a44a888c4b8bf2f79e2ce6, State: Initialized, Role: FOLLOWER
I20260810 03:24:24.904399 9184 consensus_queue.cc:260] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:24.908973 9184 ts_tablet_manager.cc:1428] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Time spent starting tablet: real 0.020s user 0.016s sys 0.001s
I20260810 03:24:24.908753 9183 raft_consensus.cc:348] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:24.909304 9183 raft_consensus.cc:374] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:24.909417 9183 raft_consensus.cc:729] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e156ed81f844facaf1cef6da27bcab0, State: Initialized, Role: FOLLOWER
I20260810 03:24:24.910122 9183 consensus_queue.cc:260] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:24.911962 9160 heartbeater.cc:503] Master 127.19.246.254:37747 was elected leader, sending a full tablet report...
I20260810 03:24:24.912487 9183 ts_tablet_manager.cc:1428] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260810 03:24:24.954118 8872 tablet.cc:2369] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:24:25.004943 9018 tablet.cc:2369] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:24:25.021318 9162 tablet.cc:2369] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:25.088866 9190 raft_consensus.cc:482] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:24:25.089258 9190 raft_consensus.cc:504] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:25.091758 9190 leader_election.cc:288] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1794ed8d6fba4ce3b3ba127fd4d351e2 (127.19.246.194:45743), d88de99b95a44a888c4b8bf2f79e2ce6 (127.19.246.193:46803)
I20260810 03:24:25.099709 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:25.092018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:50677 (local address 127.19.246.194:45743)
0810 03:24:25.092277 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:25.092285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:25.092311 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:24:25.092540 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:25.092548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:25.092664 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:25.092802 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:25.093474 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.094721 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.096881 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.098243 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.098386 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:25.098458 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:25.099216 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 03:24:25.099441 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260810 03:24:25.099710 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:25.091826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:45743 (local address 127.19.246.195:50677)
0810 03:24:25.092136 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:25.092164 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:25.092326 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:25.092889 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:25.092902 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:25.092950 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:25.093209 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:25.093229 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.094902 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.094908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:25.096676 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:25.096690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.098515 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.098523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:25.098762 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:25.098968 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 03:24:25.099100 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 03:24:25.099311 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260810 03:24:25.101132 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:25.092346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:51167 (local address 127.19.246.193:46803)
0810 03:24:25.092572 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:25.092577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:25.093403 (+ 826us) server_negotiation.cc:407] Connection header received
0810 03:24:25.093598 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:25.093603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:25.093683 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:25.093795 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:25.094514 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.095625 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.098192 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.100055 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.100260 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:25.100346 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:25.100822 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 03:24:25.100946 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:24:25.101307 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:25.092308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:46803 (local address 127.19.246.195:51167)
0810 03:24:25.093235 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:25.093256 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:25.093424 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:25.093837 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:25.093842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:25.093874 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:25.094297 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:25.094311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.095809 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.095813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:25.097789 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:25.097808 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.100287 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.100295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:25.100568 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:25.100612 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 03:24:25.100743 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:24:25.100914 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":480,"threads_started":1}
I20260810 03:24:25.101131 8967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ace1a32c5644d98e363f9519e52a3d" candidate_uuid: "2e156ed81f844facaf1cef6da27bcab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" is_pre_election: true
I20260810 03:24:25.101900 8967 raft_consensus.cc:2454] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e156ed81f844facaf1cef6da27bcab0 in term 0.
I20260810 03:24:25.102475 8825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ace1a32c5644d98e363f9519e52a3d" candidate_uuid: "2e156ed81f844facaf1cef6da27bcab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" is_pre_election: true
I20260810 03:24:25.102722 9036 leader_election.cc:302] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1794ed8d6fba4ce3b3ba127fd4d351e2, 2e156ed81f844facaf1cef6da27bcab0; no voters:
I20260810 03:24:25.103140 8825 raft_consensus.cc:2454] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e156ed81f844facaf1cef6da27bcab0 in term 0.
I20260810 03:24:25.103204 9190 raft_consensus.cc:2781] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:24:25.103416 9190 raft_consensus.cc:482] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:24:25.103569 9190 raft_consensus.cc:3037] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:25.106383 9190 raft_consensus.cc:504] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:25.107250 9190 leader_election.cc:288] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [CANDIDATE]: Term 1 election: Requested vote from peers 1794ed8d6fba4ce3b3ba127fd4d351e2 (127.19.246.194:45743), d88de99b95a44a888c4b8bf2f79e2ce6 (127.19.246.193:46803)
I20260810 03:24:25.108057 8967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ace1a32c5644d98e363f9519e52a3d" candidate_uuid: "2e156ed81f844facaf1cef6da27bcab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2"
I20260810 03:24:25.108111 8825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ace1a32c5644d98e363f9519e52a3d" candidate_uuid: "2e156ed81f844facaf1cef6da27bcab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6"
I20260810 03:24:25.108364 8967 raft_consensus.cc:3037] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:25.108379 8825 raft_consensus.cc:3037] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:25.110872 8825 raft_consensus.cc:2454] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e156ed81f844facaf1cef6da27bcab0 in term 1.
I20260810 03:24:25.110890 8967 raft_consensus.cc:2454] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e156ed81f844facaf1cef6da27bcab0 in term 1.
I20260810 03:24:25.111511 9041 leader_election.cc:302] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e156ed81f844facaf1cef6da27bcab0, d88de99b95a44a888c4b8bf2f79e2ce6; no voters:
I20260810 03:24:25.111894 9190 raft_consensus.cc:2781] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:25.112483 9190 raft_consensus.cc:686] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 1 LEADER]: Becoming Leader. State: Replica: 2e156ed81f844facaf1cef6da27bcab0, State: Running, Role: LEADER
I20260810 03:24:25.113032 9190 consensus_queue.cc:237] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } }
I20260810 03:24:25.118708 8671 catalog_manager.cc:5495] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e156ed81f844facaf1cef6da27bcab0 (127.19.246.195). New cstate: current_term: 1 leader_uuid: "2e156ed81f844facaf1cef6da27bcab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e156ed81f844facaf1cef6da27bcab0" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 41575 } health_report { overall_health: HEALTHY } } }
I20260810 03:24:25.166579 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:25.160430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.19.246.195:41575)
0810 03:24:25.160691 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:25.160697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:25.160761 (+ 64us) server_negotiation.cc:407] Connection header received
0810 03:24:25.161093 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:25.161098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:25.161199 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:25.161374 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:24:25.161797 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.162559 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.164154 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.164603 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.164748 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:25.165380 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:24:25.165909 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:24:25.166127 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:25.166285 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 03:24:25.166335 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:24:25.175778 8967 raft_consensus.cc:1261] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 1 FOLLOWER]: Refusing update from remote peer 2e156ed81f844facaf1cef6da27bcab0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:25.175777 8825 raft_consensus.cc:1261] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 1 FOLLOWER]: Refusing update from remote peer 2e156ed81f844facaf1cef6da27bcab0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:25.176896 9192 consensus_queue.cc:1035] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1794ed8d6fba4ce3b3ba127fd4d351e2" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:25.177295 9190 consensus_queue.cc:1035] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88de99b95a44a888c4b8bf2f79e2ce6" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:25.209487 8670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:25.209889 8670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:25.214897 8670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ca21877e2f40a2be087e70fc5c2a40: Sending DeleteTablet for 3 replicas of tablet a1ace1a32c5644d98e363f9519e52a3d
I20260810 03:24:25.216524 8803 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ace1a32c5644d98e363f9519e52a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:25 UTC) from {username='slave'} at 127.0.0.1:56254
I20260810 03:24:25.216532 8944 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ace1a32c5644d98e363f9519e52a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:25 UTC) from {username='slave'} at 127.0.0.1:57820
I20260810 03:24:25.216591 9090 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ace1a32c5644d98e363f9519e52a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:25 UTC) from {username='slave'} at 127.0.0.1:39528
I20260810 03:24:25.222597 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 8735
I20260810 03:24:25.227815 9211 tablet_replica.cc:330] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: stopping tablet replica
I20260810 03:24:25.228536 9211 raft_consensus.cc:2229] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:24:25.229297 9207 tablet_replica.cc:330] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: stopping tablet replica
I20260810 03:24:25.229509 9211 raft_consensus.cc:2258] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:25.230047 9210 tablet_replica.cc:330] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: stopping tablet replica
I20260810 03:24:25.230605 9210 raft_consensus.cc:2229] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:25.231150 9207 raft_consensus.cc:2229] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:25.231837 9210 raft_consensus.cc:2258] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:25.232364 9207 raft_consensus.cc:2258] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:25.232882 9211 ts_tablet_manager.cc:1905] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:25.235109 9207 ts_tablet_manager.cc:1905] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:25.235109 9210 ts_tablet_manager.cc:1905] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:25.242482 9211 ts_tablet_manager.cc:1918] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:25.242835 9211 log.cc:1196] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/a1ace1a32c5644d98e363f9519e52a3d
I20260810 03:24:25.243536 9207 ts_tablet_manager.cc:1918] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:25.243571 9211 ts_tablet_manager.cc:1939] T a1ace1a32c5644d98e363f9519e52a3d P 2e156ed81f844facaf1cef6da27bcab0: Deleting consensus metadata
I20260810 03:24:25.243839 9207 log.cc:1196] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/a1ace1a32c5644d98e363f9519e52a3d
I20260810 03:24:25.244500 9207 ts_tablet_manager.cc:1939] T a1ace1a32c5644d98e363f9519e52a3d P 1794ed8d6fba4ce3b3ba127fd4d351e2: Deleting consensus metadata
I20260810 03:24:25.245899 8654 catalog_manager.cc:4841] TS 2e156ed81f844facaf1cef6da27bcab0 (127.19.246.195:41575): tablet a1ace1a32c5644d98e363f9519e52a3d (table test-table [id=237459380854463bbf0609f861a55113]) successfully deleted
I20260810 03:24:25.246524 8652 catalog_manager.cc:4841] TS 1794ed8d6fba4ce3b3ba127fd4d351e2 (127.19.246.194:45743): tablet a1ace1a32c5644d98e363f9519e52a3d (table test-table [id=237459380854463bbf0609f861a55113]) successfully deleted
I20260810 03:24:25.249526 9210 ts_tablet_manager.cc:1918] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:25.249756 9210 log.cc:1196] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/a1ace1a32c5644d98e363f9519e52a3d
I20260810 03:24:25.250435 9210 ts_tablet_manager.cc:1939] T a1ace1a32c5644d98e363f9519e52a3d P d88de99b95a44a888c4b8bf2f79e2ce6: Deleting consensus metadata
I20260810 03:24:25.252811 8655 catalog_manager.cc:4841] TS d88de99b95a44a888c4b8bf2f79e2ce6 (127.19.246.193:46803): tablet a1ace1a32c5644d98e363f9519e52a3d (table test-table [id=237459380854463bbf0609f861a55113]) successfully deleted
I20260810 03:24:25.253257 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:25.224362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.19.246.193:46803)
0810 03:24:25.224574 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:25.224583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:25.243701 (+ 19118us) server_negotiation.cc:407] Connection header received
0810 03:24:25.244080 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:25.244086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:25.244220 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:25.244314 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:25.245109 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.246313 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.248601 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.249332 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.249490 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:25.251208 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:25.251230 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:25.251269 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:25.252165 (+ 896us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:25.252234 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:25.252263 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:25.252363 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:25.252662 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 03:24:25.252819 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 03:24:25.254441 8866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:25.350266 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 8735
I20260810 03:24:25.373179 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 8875
I20260810 03:24:25.381995 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:25.374225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.19.246.194:45743)
0810 03:24:25.374437 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:25.374443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:25.374464 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:24:25.374709 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:25.374714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:25.374880 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:25.374974 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:25.375549 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.376345 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.378463 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.378941 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.379043 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:25.380460 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:25.380470 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:25.380486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:25.381134 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:25.381186 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:25.381208 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:25.381315 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:25.381487 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 03:24:25.381586 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 03:24:25.382946 9010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:25.512657 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 8875
I20260810 03:24:25.534669 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 9021
I20260810 03:24:25.544348 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:25.535728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.19.246.195:41575)
0810 03:24:25.535940 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:25.535949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:25.536017 (+ 68us) server_negotiation.cc:407] Connection header received
0810 03:24:25.536096 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:25.536119 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:25.536284 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:25.536472 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:25.536988 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.538431 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.540000 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:25.540792 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:25.540959 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:25.542272 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:25.542288 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:25.542309 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:25.543227 (+ 918us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:25.543301 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:25.543331 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:25.543529 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:25.543768 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 03:24:25.543893 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:24:25.545349 9155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:25.672803 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 9021
I20260810 03:24:25.693602 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 8636
I20260810 03:24:25.694861 8703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:25.806918 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 8636
2026-08-10T03:24:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3334 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:25 dist-test-slave-btw8 krb5kdc[9219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:25 dist-test-slave-btw8 krb5kdc[9219](info): set up 2 sockets
Aug 10 03:24:25 dist-test-slave-btw8 krb5kdc[9219](info): commencing operation
krb5kdc: starting...
W20260810 03:24:27.053222 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:27 dist-test-slave-btw8 krb5kdc[9219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:24:27.431702 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:43535
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:43535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:27.809906 9233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:27.810348 9233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:27.810530 9233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:27.818755 9233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:27.818980 9233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:27.819108 9233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:27.819224 9233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:27.819314 9233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:27.831794 9233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:43535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:43535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:27.833549 9233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:27.835121 9233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:27.849864 9239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:27.849893 9242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:27.849798 9240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:27.851007 9233 server_base.cc:1034] running on GCE node
I20260810 03:24:27.852102 9233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:27.853963 9233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:27.855279 9233 hybrid_clock.cc:648] HybridClock initialized: now 1786332267855208 us; error 86 us; skew 500 ppm
I20260810 03:24:27.855780 9233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:27 dist-test-slave-btw8 krb5kdc[9219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332267, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:27.860003 9233 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:24:27.862313 9233 webserver.cc:460] Webserver started at http://127.19.246.254:38483/ using document root <none> and password file <none>
I20260810 03:24:27.863169 9233 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:27.863399 9233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:27.863787 9233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:27.867324 9233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "30a9be911cb142cda08a4e4ff3a73087"
format_stamp: "Formatted at 2026-08-10 03:24:27 on dist-test-slave-btw8"
server_key: "2fcca13983984026d8e8b6430dba46be"
server_key_iv: "1df37912954575e60ce347f5486a4676"
server_key_version: "encryptionkey@0"
I20260810 03:24:27.868350 9233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "30a9be911cb142cda08a4e4ff3a73087"
format_stamp: "Formatted at 2026-08-10 03:24:27 on dist-test-slave-btw8"
server_key: "2fcca13983984026d8e8b6430dba46be"
server_key_iv: "1df37912954575e60ce347f5486a4676"
server_key_version: "encryptionkey@0"
I20260810 03:24:27.875000 9233 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 03:24:27.880311 9250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:27.881842 9233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:24:27.882079 9233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "30a9be911cb142cda08a4e4ff3a73087"
format_stamp: "Formatted at 2026-08-10 03:24:27 on dist-test-slave-btw8"
server_key: "2fcca13983984026d8e8b6430dba46be"
server_key_iv: "1df37912954575e60ce347f5486a4676"
server_key_version: "encryptionkey@0"
I20260810 03:24:27.882323 9233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:27.903625 9233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:27.909307 9233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:27.909650 9233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:27.930783 9233 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:43535
I20260810 03:24:27.930822 9308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:43535 every 8 connection(s)
I20260810 03:24:27.932430 9233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:27.937206 9309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:27.940366 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 9233
I20260810 03:24:27.940601 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:27.941046 20443 external_mini_cluster.cc:1428] Setting key 05e68b13a9b26a0cf2c29c6927906c94
I20260810 03:24:27.950984 9309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087: Bootstrap starting.
I20260810 03:24:27.955600 9309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:27.957165 9309 log.cc:824] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:24:27 dist-test-slave-btw8 krb5kdc[9219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:27.960983 9309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087: No bootstrap required, opened a new log
I20260810 03:24:27.969116 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:27.943275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.19.246.254:43535)
0810 03:24:27.944269 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:27.944283 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:27.944332 (+ 49us) server_negotiation.cc:407] Connection header received
0810 03:24:27.946168 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:27.946194 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:27.946752 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:27.947129 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:27.949296 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:27.950774 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:27.952806 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:27.953686 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:27.953871 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:27.960067 (+ 6196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:27.960077 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:27.960094 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:27.960141 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:27.963899 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:27.965009 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:27.965018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:27.965026 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:27.965262 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:27.965799 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:27.965803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:27.965806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:27.966515 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:27.966868 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:27.967309 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 03:24:27.967944 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":463,"threads_started":1}
I20260810 03:24:27.970741 9309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a9be911cb142cda08a4e4ff3a73087" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43535 } }
I20260810 03:24:27.971201 9309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:27.971392 9309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a9be911cb142cda08a4e4ff3a73087, State: Initialized, Role: FOLLOWER
I20260810 03:24:27.972113 9309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a9be911cb142cda08a4e4ff3a73087" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43535 } }
I20260810 03:24:27.972396 9309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:27.972540 9309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:27.972769 9309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:27.975112 9309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a9be911cb142cda08a4e4ff3a73087" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43535 } }
I20260810 03:24:27.975741 9309 leader_election.cc:302] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30a9be911cb142cda08a4e4ff3a73087; no voters:
I20260810 03:24:27.976496 9309 leader_election.cc:288] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:27.976649 9314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:27.977700 9314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [term 1 LEADER]: Becoming Leader. State: Replica: 30a9be911cb142cda08a4e4ff3a73087, State: Running, Role: LEADER
I20260810 03:24:27.978330 9314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a9be911cb142cda08a4e4ff3a73087" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43535 } }
I20260810 03:24:27.979192 9309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:27.982234 9315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30a9be911cb142cda08a4e4ff3a73087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a9be911cb142cda08a4e4ff3a73087" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43535 } } }
I20260810 03:24:27.982950 9315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:27.982791 9316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a9be911cb142cda08a4e4ff3a73087. Latest consensus state: current_term: 1 leader_uuid: "30a9be911cb142cda08a4e4ff3a73087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a9be911cb142cda08a4e4ff3a73087" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 43535 } } }
I20260810 03:24:27.983086 9316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:27.984190 9323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:27.990722 9323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:28.001971 9323 catalog_manager.cc:1347] Generated new cluster ID: 00db30f4167a49718931ad19202598d7
I20260810 03:24:28.002143 9323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:28.029150 9323 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:28.031307 9323 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:28.041630 9323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087: Generated new TSK 0
I20260810 03:24:28.042896 9323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.246.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
Entry for principal kudu/127.19.246.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab.
I20260810 03:24:28.140421 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43535
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:28.533589 9338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:28.533901 9338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:28.533999 9338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:28.541548 9338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:28.541698 9338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:28.541878 9338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:24:28.552220 9338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.193.keytab
--principal=kudu/127.19.246.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:28.553670 9338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:28.555107 9338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:28.569980 9346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:28.569919 9348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:28.570013 9343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:28.571208 9338 server_base.cc:1034] running on GCE node
I20260810 03:24:28.572170 9338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:28.573366 9338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:28.574636 9338 hybrid_clock.cc:648] HybridClock initialized: now 1786332268574586 us; error 40 us; skew 500 ppm
I20260810 03:24:28.575030 9338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:28 dist-test-slave-btw8 krb5kdc[9219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332268, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:28.578796 9338 init.cc:377] Logged in from keytab as kudu/127.19.246.193@KRBTEST.COM (short username kudu)
I20260810 03:24:28.580787 9338 webserver.cc:460] Webserver started at http://127.19.246.193:34633/ using document root <none> and password file <none>
I20260810 03:24:28.581573 9338 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:28.581736 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:28.582094 9338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:28.585482 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "8947ebf77b494df1b75f14ce4c3d31c2"
format_stamp: "Formatted at 2026-08-10 03:24:28 on dist-test-slave-btw8"
server_key: "0800adbb95c4bbfcf6756686f5b4829a"
server_key_iv: "59ea6471f862532df08268872d2b986b"
server_key_version: "encryptionkey@0"
I20260810 03:24:28.586409 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "8947ebf77b494df1b75f14ce4c3d31c2"
format_stamp: "Formatted at 2026-08-10 03:24:28 on dist-test-slave-btw8"
server_key: "0800adbb95c4bbfcf6756686f5b4829a"
server_key_iv: "59ea6471f862532df08268872d2b986b"
server_key_version: "encryptionkey@0"
I20260810 03:24:28.592352 9338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 03:24:28.596553 9356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:28.598162 9338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 03:24:28.598379 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "8947ebf77b494df1b75f14ce4c3d31c2"
format_stamp: "Formatted at 2026-08-10 03:24:28 on dist-test-slave-btw8"
server_key: "0800adbb95c4bbfcf6756686f5b4829a"
server_key_iv: "59ea6471f862532df08268872d2b986b"
server_key_version: "encryptionkey@0"
I20260810 03:24:28.598623 9338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:28.613420 9338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:28.618355 9338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:28.618717 9338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:28.620025 9338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:28.621657 9338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:28.621783 9338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:28.621948 9338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:28.622032 9338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:28.659454 9338 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:34397
I20260810 03:24:28.659549 9476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:34397 every 8 connection(s)
I20260810 03:24:28.660864 9338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:24:28.666950 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 9338
I20260810 03:24:28.667213 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:24:28.667881 20443 external_mini_cluster.cc:1428] Setting key 222a8791bfee91d6dc5f4cacdf9ea8b0
Aug 10 03:24:28 dist-test-slave-btw8 krb5kdc[9219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332268, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.193@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:28.688457 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:28.663882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:48931 (local address 127.19.246.254:43535)
0810 03:24:28.664427 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:28.664436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:28.665464 (+ 1028us) server_negotiation.cc:407] Connection header received
0810 03:24:28.667679 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:28.667687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:28.667790 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:28.667961 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:28.669602 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:28.670632 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:28.673464 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:28.673963 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:28.674115 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:28.679462 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:28.679472 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:28.679480 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:28.679536 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:28.683814 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:28.685145 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:28.685155 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:28.685161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:28.685348 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:28.686085 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:28.686089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:28.686091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:28.686465 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:28.686587 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:28.687947 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0810 03:24:28.688167 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260810 03:24:28.690151 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:28.664503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43535 (local address 127.19.246.193:48931)
0810 03:24:28.665282 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:28.665322 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:28.667317 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:28.668211 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:28.668221 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:28.668813 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:28.669422 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:28.669440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:28.670830 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:28.670838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:28.673257 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:28.673278 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:28.674270 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:28.674280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:28.674637 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:28.675592 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:28.675627 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:28.679100 (+ 3473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:28.684069 (+ 4969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:28.684081 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:28.684101 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:28.684972 (+ 871us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:28.685594 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:28.685601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:28.685607 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:28.685947 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:28.686629 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:28.686647 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:28.687250 (+ 603us) client_negotiation.cc:769] Sending connection context
0810 03:24:28.687651 (+ 401us) client_negotiation.cc:240] Negotiation successful
0810 03:24:28.688415 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":309,"threads_started":1}
I20260810 03:24:28.692116 9477 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43535
I20260810 03:24:28.692471 9477 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:28.693279 9477 heartbeater.cc:511] Master 127.19.246.254:43535 requested a full tablet report, sending...
I20260810 03:24:28.695695 9269 ts_manager.cc:194] Registered new tserver with Master: 8947ebf77b494df1b75f14ce4c3d31c2 (127.19.246.193:34397)
I20260810 03:24:28.697265 9269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.193@KRBTEST.COM'} at 127.19.246.193:48931
WARNING: no policy specified for kudu/127.19.246.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
Entry for principal kudu/127.19.246.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab.
I20260810 03:24:28.752995 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43535
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:29.134887 9484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:29.135248 9484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:29.135424 9484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:29.143997 9484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:29.144140 9484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:29.144340 9484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:24:29.154898 9484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.194.keytab
--principal=kudu/127.19.246.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:29.156387 9484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:29.157846 9484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:29.174391 9490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:29.174445 9492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:29.174459 9489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:29.175671 9484 server_base.cc:1034] running on GCE node
I20260810 03:24:29.176595 9484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:29.177855 9484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:29.179121 9484 hybrid_clock.cc:648] HybridClock initialized: now 1786332269179036 us; error 78 us; skew 500 ppm
I20260810 03:24:29.179548 9484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:29 dist-test-slave-btw8 krb5kdc[9219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332269, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:29.188103 9484 init.cc:377] Logged in from keytab as kudu/127.19.246.194@KRBTEST.COM (short username kudu)
I20260810 03:24:29.189905 9484 webserver.cc:460] Webserver started at http://127.19.246.194:45343/ using document root <none> and password file <none>
I20260810 03:24:29.190761 9484 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:29.190943 9484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:29.191497 9484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:29.194968 9484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "2ef20a7adb8f445e840dee40834329f3"
format_stamp: "Formatted at 2026-08-10 03:24:29 on dist-test-slave-btw8"
server_key: "ac69ef3eb4a00edaa3fbda1321f8a590"
server_key_iv: "3db480785c84d6167dd8cf571fc8b88b"
server_key_version: "encryptionkey@0"
I20260810 03:24:29.195974 9484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "2ef20a7adb8f445e840dee40834329f3"
format_stamp: "Formatted at 2026-08-10 03:24:29 on dist-test-slave-btw8"
server_key: "ac69ef3eb4a00edaa3fbda1321f8a590"
server_key_iv: "3db480785c84d6167dd8cf571fc8b88b"
server_key_version: "encryptionkey@0"
I20260810 03:24:29.202265 9484 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:24:29.207127 9500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:29.208868 9484 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:24:29.209097 9484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "2ef20a7adb8f445e840dee40834329f3"
format_stamp: "Formatted at 2026-08-10 03:24:29 on dist-test-slave-btw8"
server_key: "ac69ef3eb4a00edaa3fbda1321f8a590"
server_key_iv: "3db480785c84d6167dd8cf571fc8b88b"
server_key_version: "encryptionkey@0"
I20260810 03:24:29.209342 9484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:29.232820 9484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:29.237960 9484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:29.238359 9484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:29.239583 9484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:29.241326 9484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:29.241447 9484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:29.241613 9484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:29.241703 9484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:29.281286 9484 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:46311
I20260810 03:24:29.281353 9619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:46311 every 8 connection(s)
I20260810 03:24:29.282959 9484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:24:29.290206 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 9484
I20260810 03:24:29.290459 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:24:29.290947 20443 external_mini_cluster.cc:1428] Setting key 8643c5149e8a24f089d1f0390bd28fba
Aug 10 03:24:29 dist-test-slave-btw8 krb5kdc[9219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332269, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.194@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:29.305693 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.286176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:55243 (local address 127.19.246.254:43535)
0810 03:24:29.286778 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:29.286785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:29.287883 (+ 1098us) server_negotiation.cc:407] Connection header received
0810 03:24:29.289892 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:29.289898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:29.290007 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:29.290134 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:29.292064 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.293088 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.294775 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.295225 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.295368 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:29.299139 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:29.299144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:29.299148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:29.299185 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:29.302115 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:29.303114 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:29.303119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:29.303121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:29.303254 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:29.303760 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:29.303770 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:29.303772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:29.304069 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:29.304240 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:29.305009 (+ 769us) server_negotiation.cc:299] Negotiation successful
0810 03:24:29.305270 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
I20260810 03:24:29.306748 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.286734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43535 (local address 127.19.246.194:55243)
0810 03:24:29.287686 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:29.287743 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:29.289549 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:29.290436 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:29.290445 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:29.291134 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:29.291870 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:29.291889 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.293262 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.293267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:29.294608 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:29.294622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.295500 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.295508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:29.295762 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:29.296521 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:29.296544 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:29.298861 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:29.302287 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:29.302298 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:29.302319 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:29.302958 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:29.303391 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:29.303396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:29.303400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:29.303604 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:29.304208 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:29.304216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:29.304611 (+ 395us) client_negotiation.cc:769] Sending connection context
0810 03:24:29.304895 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 03:24:29.305499 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":349,"threads_started":1}
I20260810 03:24:29.308408 9620 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43535
I20260810 03:24:29.308812 9620 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:29.309757 9620 heartbeater.cc:511] Master 127.19.246.254:43535 requested a full tablet report, sending...
I20260810 03:24:29.311585 9269 ts_manager.cc:194] Registered new tserver with Master: 2ef20a7adb8f445e840dee40834329f3 (127.19.246.194:46311)
I20260810 03:24:29.312557 9269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.194@KRBTEST.COM'} at 127.19.246.194:55243
WARNING: no policy specified for kudu/127.19.246.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
Entry for principal kudu/127.19.246.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab.
I20260810 03:24:29.372423 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43535
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:24:29.701151 9477 heartbeater.cc:503] Master 127.19.246.254:43535 was elected leader, sending a full tablet report...
W20260810 03:24:29.740736 9628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:29.741070 9628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:29.741174 9628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:29.749049 9628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:29.749181 9628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:29.749297 9628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:24:29.759802 9628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.195.keytab
--principal=kudu/127.19.246.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:43535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:29.761188 9628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:29.762552 9628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:29.777667 9635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:29.777653 9634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:29.778301 9637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:29.778735 9628 server_base.cc:1034] running on GCE node
I20260810 03:24:29.779814 9628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:29.781178 9628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:29.782405 9628 hybrid_clock.cc:648] HybridClock initialized: now 1786332269782333 us; error 69 us; skew 500 ppm
I20260810 03:24:29.782810 9628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:29 dist-test-slave-btw8 krb5kdc[9219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332269, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:29.787930 9628 init.cc:377] Logged in from keytab as kudu/127.19.246.195@KRBTEST.COM (short username kudu)
I20260810 03:24:29.789850 9628 webserver.cc:460] Webserver started at http://127.19.246.195:35317/ using document root <none> and password file <none>
I20260810 03:24:29.790668 9628 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:29.790829 9628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:29.791173 9628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:29.794377 9628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b"
format_stamp: "Formatted at 2026-08-10 03:24:29 on dist-test-slave-btw8"
server_key: "28442a223a454b3389e805b28bdee6eb"
server_key_iv: "2ae17eed08bbbcf369f9558cd5b5b7cb"
server_key_version: "encryptionkey@0"
I20260810 03:24:29.795202 9628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b"
format_stamp: "Formatted at 2026-08-10 03:24:29 on dist-test-slave-btw8"
server_key: "28442a223a454b3389e805b28bdee6eb"
server_key_iv: "2ae17eed08bbbcf369f9558cd5b5b7cb"
server_key_version: "encryptionkey@0"
I20260810 03:24:29.801234 9628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260810 03:24:29.805830 9645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:29.807873 9628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:24:29.808132 9628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b"
format_stamp: "Formatted at 2026-08-10 03:24:29 on dist-test-slave-btw8"
server_key: "28442a223a454b3389e805b28bdee6eb"
server_key_iv: "2ae17eed08bbbcf369f9558cd5b5b7cb"
server_key_version: "encryptionkey@0"
I20260810 03:24:29.808408 9628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:29.840615 9628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:29.846256 9628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:29.846654 9628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:29.847821 9628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:29.849390 9628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:29.849466 9628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:29.849560 9628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:29.849597 9628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:29.888252 9628 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:36173
I20260810 03:24:29.888324 9767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:36173 every 8 connection(s)
I20260810 03:24:29.889787 9628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:24:29.891589 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 9628
I20260810 03:24:29.891778 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:24:29.892259 20443 external_mini_cluster.cc:1428] Setting key 026e0008106f6119a3c22f98a1f4ccc1
Aug 10 03:24:29 dist-test-slave-btw8 krb5kdc[9219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332269, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.195@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:24:29.918815 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.893151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:38659 (local address 127.19.246.254:43535)
0810 03:24:29.893731 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:29.893737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:29.895012 (+ 1275us) server_negotiation.cc:407] Connection header received
0810 03:24:29.897323 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:29.897329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:29.897438 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:29.897576 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:29.899443 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.900973 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.904515 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.905260 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.905438 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:29.911303 (+ 5865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:29.911310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:29.911315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:29.911400 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:29.914910 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:29.915995 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:29.916000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:29.916003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:29.916195 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:29.916751 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:29.916755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:29.916757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:29.917071 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:29.917211 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:29.918201 (+ 990us) server_negotiation.cc:299] Negotiation successful
0810 03:24:29.918433 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":295,"threads_started":1}
I20260810 03:24:29.919799 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.893811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:43535 (local address 127.19.246.195:38659)
0810 03:24:29.894858 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:29.894900 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:29.897016 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:29.897866 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:29.897875 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:29.898508 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:29.899191 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:29.899212 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.901239 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.901248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:29.904248 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:29.904282 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.905583 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.905592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:29.906022 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:29.906991 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:24:29.907027 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:29.911007 (+ 3980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:29.915107 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:29.915116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:29.915132 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:29.915799 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:29.916343 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:24:29.916346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:24:29.916349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:24:29.916572 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:24:29.917227 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:29.917234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:29.917635 (+ 401us) client_negotiation.cc:769] Sending connection context
0810 03:24:29.917901 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 03:24:29.918410 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":398,"threads_started":1}
I20260810 03:24:29.921651 9768 heartbeater.cc:347] Connected to a master server at 127.19.246.254:43535
I20260810 03:24:29.922046 9768 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:29.923010 9768 heartbeater.cc:511] Master 127.19.246.254:43535 requested a full tablet report, sending...
I20260810 03:24:29.924894 9269 ts_manager.cc:194] Registered new tserver with Master: 257b0dbd2b204cbdb64fa3cdcacbbc3b (127.19.246.195:36173)
I20260810 03:24:29.925741 9269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.246.195@KRBTEST.COM'} at 127.19.246.195:38659
I20260810 03:24:29.937466 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:24:29.957943 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.942644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.19.246.254:43535)
0810 03:24:29.943016 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:29.943024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:29.943388 (+ 364us) server_negotiation.cc:407] Connection header received
0810 03:24:29.943659 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:29.943664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:29.943762 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:29.943856 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:29.945519 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.946321 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.947823 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:29.948349 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:29.948485 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:29.951304 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:29.951309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:29.951313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:29.951384 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:29.954604 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:29.955566 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:29.955571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:29.955574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:29.955775 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:29.956205 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:29.956208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:29.956210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:29.956603 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:29.956874 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:29.957210 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 03:24:29.957503 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260810 03:24:29.962589 9269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:24:29.970820 9269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:24:30.025724 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.986119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:46311 (local address 127.0.0.1:51042)
0810 03:24:30.001675 (+ 15556us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:30.001704 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:30.001883 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:30.016869 (+ 14986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:30.016877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:30.016918 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:30.017537 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.017557 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.019200 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.019207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.022723 (+ 3516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.022743 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.024451 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.024459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.024725 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.024790 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 03:24:30.024982 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 03:24:30.025223 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15326,"mutex_wait_us":205,"thread_start_us":15105,"threads_started":1}
I20260810 03:24:30.025811 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.985937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36173 (local address 127.0.0.1:36574)
0810 03:24:29.997682 (+ 11745us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:29.997731 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:29.997951 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:30.016435 (+ 18484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:30.016443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:30.016495 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:30.017024 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.017042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.019380 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.019389 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.022723 (+ 3334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.022743 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.024976 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.024982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.025162 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.025192 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:24:30.025284 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 03:24:30.025427 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11448,"mutex_wait_us":213,"thread_start_us":11274,"threads_started":1}
I20260810 03:24:30.025796 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.986768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.19.246.195:36173)
0810 03:24:30.015380 (+ 28612us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.015396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.015450 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:24:30.015616 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.015629 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.016058 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.016416 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:30.017270 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.019185 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.022955 (+ 3770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.024711 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.024971 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.025156 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.025347 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.025501 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28299,"thread_start_us":28082,"threads_started":1}
I20260810 03:24:30.026898 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.986357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.19.246.194:46311)
0810 03:24:30.016259 (+ 29902us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.016267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.016298 (+ 31us) server_negotiation.cc:407] Connection header received
0810 03:24:30.016403 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.016410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.016592 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.017817 (+ 1225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:30.017999 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.019012 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.022979 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.024213 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.024431 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.024659 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.026403 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.026599 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29700,"thread_start_us":29572,"threads_started":1}
I20260810 03:24:30.027513 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.987519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.19.246.193:34397)
0810 03:24:30.014581 (+ 27062us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.014598 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.016206 (+ 1608us) server_negotiation.cc:407] Connection header received
0810 03:24:30.016339 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.016346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.016602 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.016813 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:30.017842 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.019624 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.022808 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.024711 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.024977 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.025145 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.026840 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.027052 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26780,"thread_start_us":26524,"threads_started":1}
I20260810 03:24:30.031033 9698 tablet_service.cc:1467] Processing CreateTablet for tablet cdb6dbe4a57c4f479c9b1745f2a347dd (DEFAULT_TABLE table=test-table [id=698f5396f59f4c39a19832ad774ec171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:30.031033 9550 tablet_service.cc:1467] Processing CreateTablet for tablet cdb6dbe4a57c4f479c9b1745f2a347dd (DEFAULT_TABLE table=test-table [id=698f5396f59f4c39a19832ad774ec171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:30.031863 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:29.986660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:34397 (local address 127.0.0.1:45082)
0810 03:24:30.015771 (+ 29111us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:30.015798 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:30.016008 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:30.017131 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:30.017136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:30.017169 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:30.017624 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.017633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.020164 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.020170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.022545 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.022568 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.026463 (+ 3895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.026469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.026607 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.026641 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:24:30.026778 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 03:24:30.026934 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28964,"mutex_wait_us":45,"thread_start_us":28746,"threads_started":1}
I20260810 03:24:30.032930 9698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb6dbe4a57c4f479c9b1745f2a347dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:30.032930 9550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb6dbe4a57c4f479c9b1745f2a347dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:30.039230 9406 tablet_service.cc:1467] Processing CreateTablet for tablet cdb6dbe4a57c4f479c9b1745f2a347dd (DEFAULT_TABLE table=test-table [id=698f5396f59f4c39a19832ad774ec171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:30.041849 9406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb6dbe4a57c4f479c9b1745f2a347dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:30.052186 9792 tablet_bootstrap.cc:492] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Bootstrap starting.
I20260810 03:24:30.052472 9791 tablet_bootstrap.cc:492] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Bootstrap starting.
I20260810 03:24:30.057027 9791 tablet_bootstrap.cc:654] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:30.057204 9792 tablet_bootstrap.cc:654] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:30.058409 9791 log.cc:824] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:30.058676 9792 log.cc:824] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:30.059278 9793 tablet_bootstrap.cc:492] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Bootstrap starting.
I20260810 03:24:30.063776 9791 tablet_bootstrap.cc:492] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: No bootstrap required, opened a new log
I20260810 03:24:30.063995 9792 tablet_bootstrap.cc:492] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: No bootstrap required, opened a new log
I20260810 03:24:30.064157 9793 tablet_bootstrap.cc:654] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:30.064205 9791 ts_tablet_manager.cc:1397] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260810 03:24:30.064330 9792 ts_tablet_manager.cc:1397] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260810 03:24:30.065797 9793 log.cc:824] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:30.070294 9793 tablet_bootstrap.cc:492] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: No bootstrap required, opened a new log
I20260810 03:24:30.070681 9793 ts_tablet_manager.cc:1397] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260810 03:24:30.074347 9791 raft_consensus.cc:348] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.074895 9791 raft_consensus.cc:374] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:30.075031 9791 raft_consensus.cc:729] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef20a7adb8f445e840dee40834329f3, State: Initialized, Role: FOLLOWER
I20260810 03:24:30.075940 9791 consensus_queue.cc:260] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.076905 9792 raft_consensus.cc:348] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.077466 9792 raft_consensus.cc:374] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:30.077476 9620 heartbeater.cc:503] Master 127.19.246.254:43535 was elected leader, sending a full tablet report...
I20260810 03:24:30.077579 9792 raft_consensus.cc:729] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257b0dbd2b204cbdb64fa3cdcacbbc3b, State: Initialized, Role: FOLLOWER
I20260810 03:24:30.078416 9791 ts_tablet_manager.cc:1428] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 03:24:30.078708 9792 consensus_queue.cc:260] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.080442 9768 heartbeater.cc:503] Master 127.19.246.254:43535 was elected leader, sending a full tablet report...
I20260810 03:24:30.081658 9792 ts_tablet_manager.cc:1428] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260810 03:24:30.084940 9793 raft_consensus.cc:348] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.085562 9793 raft_consensus.cc:374] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:30.085698 9793 raft_consensus.cc:729] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8947ebf77b494df1b75f14ce4c3d31c2, State: Initialized, Role: FOLLOWER
I20260810 03:24:30.086625 9793 consensus_queue.cc:260] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.088726 9793 ts_tablet_manager.cc:1428] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
W20260810 03:24:30.141928 9769 tablet.cc:2369] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:24:30.163929 9478 tablet.cc:2369] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:30.284792 9797 raft_consensus.cc:482] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 03:24:30.285777 9623 tablet.cc:2369] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:30.285344 9797 raft_consensus.cc:504] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.288127 9797 leader_election.cc:288] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 257b0dbd2b204cbdb64fa3cdcacbbc3b (127.19.246.195:36173), 8947ebf77b494df1b75f14ce4c3d31c2 (127.19.246.193:34397)
I20260810 03:24:30.298153 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:30.290107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:46341 (local address 127.19.246.193:34397)
0810 03:24:30.290358 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.290365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.290572 (+ 207us) server_negotiation.cc:407] Connection header received
0810 03:24:30.290753 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.290758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.290844 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.290941 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:30.291866 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.293082 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.295943 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.296899 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.297124 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.297226 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.297681 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.297873 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 03:24:30.298379 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:30.289500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:34397 (local address 127.19.246.194:46341)
0810 03:24:30.290440 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:30.290458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:30.290598 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:30.291011 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:30.291022 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:30.291050 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:30.291671 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.291686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.293257 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.293269 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.295755 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.295767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.297173 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.297180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.297451 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.297491 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:24:30.297615 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 03:24:30.297823 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":56,"thread_start_us":551,"threads_started":1}
I20260810 03:24:30.298408 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:30.288706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:33457 (local address 127.19.246.195:36173)
0810 03:24:30.288924 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.288932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.290164 (+ 1232us) server_negotiation.cc:407] Connection header received
0810 03:24:30.290527 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.290533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.290642 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.290768 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:30.291497 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.292467 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.295614 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.297064 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.297255 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.297335 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.297912 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.298082 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:24:30.299347 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:30.288349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:36173 (local address 127.19.246.194:33457)
0810 03:24:30.290022 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:30.290052 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:30.290318 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:30.290778 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:30.290783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:30.290806 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:30.291273 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.291290 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.292642 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.292648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.295384 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:30.295409 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.297461 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.297467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:30.297696 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.297728 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:24:30.298733 (+ 1005us) client_negotiation.cc:240] Negotiation successful
0810 03:24:30.298895 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1462,"mutex_wait_us":5,"thread_start_us":1174,"threads_started":1}
I20260810 03:24:30.299854 9429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb6dbe4a57c4f479c9b1745f2a347dd" candidate_uuid: "2ef20a7adb8f445e840dee40834329f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" is_pre_election: true
I20260810 03:24:30.300583 9429 raft_consensus.cc:2454] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef20a7adb8f445e840dee40834329f3 in term 0.
I20260810 03:24:30.300529 9719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb6dbe4a57c4f479c9b1745f2a347dd" candidate_uuid: "2ef20a7adb8f445e840dee40834329f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" is_pre_election: true
I20260810 03:24:30.301095 9719 raft_consensus.cc:2454] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef20a7adb8f445e840dee40834329f3 in term 0.
I20260810 03:24:30.301470 9504 leader_election.cc:302] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef20a7adb8f445e840dee40834329f3, 8947ebf77b494df1b75f14ce4c3d31c2; no voters:
I20260810 03:24:30.302013 9797 raft_consensus.cc:2781] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:24:30.302202 9797 raft_consensus.cc:482] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:24:30.302366 9797 raft_consensus.cc:3037] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:30.305029 9797 raft_consensus.cc:504] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.305721 9797 leader_election.cc:288] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [CANDIDATE]: Term 1 election: Requested vote from peers 257b0dbd2b204cbdb64fa3cdcacbbc3b (127.19.246.195:36173), 8947ebf77b494df1b75f14ce4c3d31c2 (127.19.246.193:34397)
I20260810 03:24:30.306365 9429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb6dbe4a57c4f479c9b1745f2a347dd" candidate_uuid: "2ef20a7adb8f445e840dee40834329f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8947ebf77b494df1b75f14ce4c3d31c2"
I20260810 03:24:30.306385 9719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb6dbe4a57c4f479c9b1745f2a347dd" candidate_uuid: "2ef20a7adb8f445e840dee40834329f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b"
I20260810 03:24:30.306631 9429 raft_consensus.cc:3037] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:30.306679 9719 raft_consensus.cc:3037] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:30.309335 9429 raft_consensus.cc:2454] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef20a7adb8f445e840dee40834329f3 in term 1.
I20260810 03:24:30.309525 9719 raft_consensus.cc:2454] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef20a7adb8f445e840dee40834329f3 in term 1.
I20260810 03:24:30.310144 9504 leader_election.cc:302] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 257b0dbd2b204cbdb64fa3cdcacbbc3b, 2ef20a7adb8f445e840dee40834329f3, 8947ebf77b494df1b75f14ce4c3d31c2; no voters:
I20260810 03:24:30.310629 9797 raft_consensus.cc:2781] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:30.311187 9797 raft_consensus.cc:686] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 1 LEADER]: Becoming Leader. State: Replica: 2ef20a7adb8f445e840dee40834329f3, State: Running, Role: LEADER
I20260810 03:24:30.311820 9797 consensus_queue.cc:237] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } }
I20260810 03:24:30.318174 9269 catalog_manager.cc:5495] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef20a7adb8f445e840dee40834329f3 (127.19.246.194). New cstate: current_term: 1 leader_uuid: "2ef20a7adb8f445e840dee40834329f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef20a7adb8f445e840dee40834329f3" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 46311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 } health_report { overall_health: UNKNOWN } } }
I20260810 03:24:30.374826 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:30.368721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.19.246.194:46311)
0810 03:24:30.368940 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.368947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.369055 (+ 108us) server_negotiation.cc:407] Connection header received
0810 03:24:30.369332 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.369337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.369430 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.369678 (+ 248us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:24:30.370071 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.370830 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.372344 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.372793 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.372957 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.373512 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:24:30.374128 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:24:30.374292 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.374396 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.374496 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:24:30.384724 9429 raft_consensus.cc:1261] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 1 FOLLOWER]: Refusing update from remote peer 2ef20a7adb8f445e840dee40834329f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:30.384747 9719 raft_consensus.cc:1261] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 1 FOLLOWER]: Refusing update from remote peer 2ef20a7adb8f445e840dee40834329f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:30.386050 9802 consensus_queue.cc:1035] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257b0dbd2b204cbdb64fa3cdcacbbc3b" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 36173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:30.386593 9797 consensus_queue.cc:1035] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8947ebf77b494df1b75f14ce4c3d31c2" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:30.400583 9806 mvcc.cc:205] Tried to move back new op lower bound from 7316816979483832320 to 7316816979201978368. Current Snapshot: MvccSnapshot[applied={T|T < 7316816979483832320}]
I20260810 03:24:30.408589 9807 mvcc.cc:205] Tried to move back new op lower bound from 7316816979483832320 to 7316816979201978368. Current Snapshot: MvccSnapshot[applied={T|T < 7316816979483832320 or (T in {7316816979483832320})}]
I20260810 03:24:30.416496 9269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:30.416853 9269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:30.421016 9269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30a9be911cb142cda08a4e4ff3a73087: Sending DeleteTablet for 3 replicas of tablet cdb6dbe4a57c4f479c9b1745f2a347dd
I20260810 03:24:30.422608 9698 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb6dbe4a57c4f479c9b1745f2a347dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:30 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:36574
I20260810 03:24:30.422794 9406 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb6dbe4a57c4f479c9b1745f2a347dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:30 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:45082
I20260810 03:24:30.422960 9550 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb6dbe4a57c4f479c9b1745f2a347dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:30 UTC) from {username='kudu', principal='kudu/127.19.246.254@KRBTEST.COM'} at 127.0.0.1:51042
I20260810 03:24:30.427704 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 9338
I20260810 03:24:30.435927 9818 tablet_replica.cc:330] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: stopping tablet replica
I20260810 03:24:30.436463 9818 raft_consensus.cc:2229] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:30.437047 9818 raft_consensus.cc:2258] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:30.440459 9818 ts_tablet_manager.cc:1905] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:30.452718 9818 ts_tablet_manager.cc:1918] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:30.453224 9818 log.cc:1196] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-0/wal/wals/cdb6dbe4a57c4f479c9b1745f2a347dd
I20260810 03:24:30.454245 9818 ts_tablet_manager.cc:1939] T cdb6dbe4a57c4f479c9b1745f2a347dd P 8947ebf77b494df1b75f14ce4c3d31c2: Deleting consensus metadata
I20260810 03:24:30.455516 9819 tablet_replica.cc:330] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: stopping tablet replica
I20260810 03:24:30.456131 9819 raft_consensus.cc:2229] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:24:30.456952 9819 raft_consensus.cc:2258] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:30.457756 9255 catalog_manager.cc:4841] TS 8947ebf77b494df1b75f14ce4c3d31c2 (127.19.246.193:34397): tablet cdb6dbe4a57c4f479c9b1745f2a347dd (table test-table [id=698f5396f59f4c39a19832ad774ec171]) successfully deleted
I20260810 03:24:30.459765 9817 tablet_replica.cc:330] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: stopping tablet replica
I20260810 03:24:30.460323 9817 raft_consensus.cc:2229] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:30.460886 9817 raft_consensus.cc:2258] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:30.460952 9819 ts_tablet_manager.cc:1905] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:30.463289 9817 ts_tablet_manager.cc:1905] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 03:24:30 dist-test-slave-btw8 krb5kdc[9219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.193@KRBTEST.COM
I20260810 03:24:30.470196 9819 ts_tablet_manager.cc:1918] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:30.470431 9819 log.cc:1196] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/cdb6dbe4a57c4f479c9b1745f2a347dd
I20260810 03:24:30.471056 9819 ts_tablet_manager.cc:1939] T cdb6dbe4a57c4f479c9b1745f2a347dd P 2ef20a7adb8f445e840dee40834329f3: Deleting consensus metadata
I20260810 03:24:30.472326 9817 ts_tablet_manager.cc:1918] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:30.472559 9817 log.cc:1196] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/cdb6dbe4a57c4f479c9b1745f2a347dd
I20260810 03:24:30.473271 9817 ts_tablet_manager.cc:1939] T cdb6dbe4a57c4f479c9b1745f2a347dd P 257b0dbd2b204cbdb64fa3cdcacbbc3b: Deleting consensus metadata
I20260810 03:24:30.474080 9252 catalog_manager.cc:4841] TS 2ef20a7adb8f445e840dee40834329f3 (127.19.246.194:46311): tablet cdb6dbe4a57c4f479c9b1745f2a347dd (table test-table [id=698f5396f59f4c39a19832ad774ec171]) successfully deleted
I20260810 03:24:30.475556 9251 catalog_manager.cc:4841] TS 257b0dbd2b204cbdb64fa3cdcacbbc3b (127.19.246.195:36173): tablet cdb6dbe4a57c4f479c9b1745f2a347dd (table test-table [id=698f5396f59f4c39a19832ad774ec171]) successfully deleted
I20260810 03:24:30.479307 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:30.429256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.19.246.193:34397)
0810 03:24:30.429462 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.429469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.451601 (+ 22132us) server_negotiation.cc:407] Connection header received
0810 03:24:30.452189 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.452197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.452296 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.452385 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:30.458286 (+ 5901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.459673 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.462437 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.463192 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.463331 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.470245 (+ 6914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:30.470271 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:30.470293 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:30.470365 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:30.475266 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:30.476534 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:30.476545 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:30.476547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:30.476731 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:30.477278 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:30.477283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:30.477287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:30.477735 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:30.478118 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.478495 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.478785 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:24:30.480286 9472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:30.581171 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 9338
I20260810 03:24:30.611232 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 9484
Aug 10 03:24:30 dist-test-slave-btw8 krb5kdc[9219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.194@KRBTEST.COM
I20260810 03:24:30.633610 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:30.612250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.19.246.194:46311)
0810 03:24:30.612671 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.612680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.612715 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:24:30.612779 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.612783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.612851 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.612973 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:30.614945 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.616046 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.618156 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.618743 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.618861 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.625340 (+ 6479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:30.625359 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:30.625378 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:30.625430 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:30.629807 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:30.630813 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:30.630824 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:30.630829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:30.631145 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:30.631701 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:30.631707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:30.631711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:30.632248 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:30.632619 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.632879 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.633159 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260810 03:24:30.634624 9614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:30.760066 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 9484
I20260810 03:24:30.783432 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 9628
Aug 10 03:24:30 dist-test-slave-btw8 krb5kdc[9219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.195@KRBTEST.COM
I20260810 03:24:30.802112 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:30.784267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.19.246.195:36173)
0810 03:24:30.784477 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:30.784488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:30.784591 (+ 103us) server_negotiation.cc:407] Connection header received
0810 03:24:30.784894 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:30.784902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:30.785027 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:30.785238 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:30.787120 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.788107 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.789552 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:30.790002 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:30.790134 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:30.795266 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:30.795284 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:30.795301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:30.795346 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:30.798974 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:30.799821 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:30.799829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:30.799831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:30.799971 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:30.800376 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:30.800380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:30.800382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:30.800796 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:30.801035 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:30.801402 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 03:24:30.801733 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:24:30.802866 9763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:30.898428 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 9628
I20260810 03:24:30.920897 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 9233
I20260810 03:24:30.922154 9304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:31.038272 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 9233
2026-08-10T03:24:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5230 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T03:24:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:31Z Disabled control of system clock
I20260810 03:24:31.141587 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:35109
--webserver_interface=127.19.246.254
--webserver_port=0
--builtin_ntp_servers=127.19.246.212:34419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:35109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:24:31.527221 9828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:31.527715 9828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:31.527879 9828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:31.535917 9828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:31.536072 9828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:31.536111 9828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:31.536257 9828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:31.536326 9828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:31.547161 9828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:35109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:35109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:31.548835 9828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:31.550889 9828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:31.564898 9836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:31.564815 9837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:31.565531 9839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:31.565974 9828 server_base.cc:1034] running on GCE node
I20260810 03:24:31.566955 9828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:31.568861 9828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:31.570123 9828 hybrid_clock.cc:648] HybridClock initialized: now 1786332271570068 us; error 41 us; skew 500 ppm
I20260810 03:24:31.570658 9828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:31.573974 9828 webserver.cc:460] Webserver started at http://127.19.246.254:41099/ using document root <none> and password file <none>
I20260810 03:24:31.574999 9828 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:31.575173 9828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:31.575680 9828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:31.579685 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "31c25f1b5fdf4897a0fab21093a8081e"
format_stamp: "Formatted at 2026-08-10 03:24:31 on dist-test-slave-btw8"
server_key: "4d3a436adfb9dd34a45c47697915795e"
server_key_iv: "5d28c9ba5d1becac983308302b6e2c2e"
server_key_version: "encryptionkey@0"
I20260810 03:24:31.580863 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "31c25f1b5fdf4897a0fab21093a8081e"
format_stamp: "Formatted at 2026-08-10 03:24:31 on dist-test-slave-btw8"
server_key: "4d3a436adfb9dd34a45c47697915795e"
server_key_iv: "5d28c9ba5d1becac983308302b6e2c2e"
server_key_version: "encryptionkey@0"
I20260810 03:24:31.588933 9828 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260810 03:24:31.593778 9845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:31.595763 9828 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:24:31.596066 9828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "31c25f1b5fdf4897a0fab21093a8081e"
format_stamp: "Formatted at 2026-08-10 03:24:31 on dist-test-slave-btw8"
server_key: "4d3a436adfb9dd34a45c47697915795e"
server_key_iv: "5d28c9ba5d1becac983308302b6e2c2e"
server_key_version: "encryptionkey@0"
I20260810 03:24:31.596352 9828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:31.615101 9828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:31.616928 9828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:31.617370 9828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:31.641570 9907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:35109 every 8 connection(s)
I20260810 03:24:31.641561 9828 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:35109
I20260810 03:24:31.643594 9828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:31.648471 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:31.650748 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 9828
I20260810 03:24:31.650951 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:31.651568 20443 external_mini_cluster.cc:1428] Setting key 67106940f593f71e8e766d43533f5374
I20260810 03:24:31.661659 9908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e: Bootstrap starting.
I20260810 03:24:31.665556 9908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:31.666875 9908 log.cc:824] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:31.667943 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:31.654007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.19.246.254:35109)
0810 03:24:31.655332 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:31.655429 (+ 97us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:31.655486 (+ 57us) server_negotiation.cc:407] Connection header received
0810 03:24:31.657466 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:31.657494 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:31.658124 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:31.658623 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:31.659294 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:31.660722 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:31.662294 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:31.663048 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:31.663294 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:31.664835 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:31.664846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:31.664867 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:31.665511 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:31.665594 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:31.665619 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:31.665790 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:31.666095 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 03:24:31.666470 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":399,"threads_started":1}
I20260810 03:24:31.670753 9908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e: No bootstrap required, opened a new log
I20260810 03:24:31.679469 9908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c25f1b5fdf4897a0fab21093a8081e" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35109 } }
I20260810 03:24:31.679909 9908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:31.679993 9908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c25f1b5fdf4897a0fab21093a8081e, State: Initialized, Role: FOLLOWER
I20260810 03:24:31.680610 9908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c25f1b5fdf4897a0fab21093a8081e" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35109 } }
I20260810 03:24:31.680830 9908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:31.680927 9908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:31.681063 9908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:31.683710 9908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c25f1b5fdf4897a0fab21093a8081e" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35109 } }
I20260810 03:24:31.684348 9908 leader_election.cc:302] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31c25f1b5fdf4897a0fab21093a8081e; no voters:
I20260810 03:24:31.684958 9908 leader_election.cc:288] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:31.685446 9914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:31.686334 9914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [term 1 LEADER]: Becoming Leader. State: Replica: 31c25f1b5fdf4897a0fab21093a8081e, State: Running, Role: LEADER
I20260810 03:24:31.686923 9914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c25f1b5fdf4897a0fab21093a8081e" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35109 } }
I20260810 03:24:31.687803 9908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:31.690398 9916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31c25f1b5fdf4897a0fab21093a8081e. Latest consensus state: current_term: 1 leader_uuid: "31c25f1b5fdf4897a0fab21093a8081e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c25f1b5fdf4897a0fab21093a8081e" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35109 } } }
I20260810 03:24:31.690397 9915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31c25f1b5fdf4897a0fab21093a8081e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c25f1b5fdf4897a0fab21093a8081e" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 35109 } } }
I20260810 03:24:31.691005 9915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:31.690980 9916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:31.693929 9922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:31.700026 9922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:31.713164 9922 catalog_manager.cc:1347] Generated new cluster ID: fc822a1a029e4782830b2e5f67b94abe
I20260810 03:24:31.713339 9922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:31.723680 9922 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:31.725238 9922 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:31.735699 9922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e: Generated new TSK 0
I20260810 03:24:31.736729 9922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:31.748045 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.193:0
--local_ip_for_outbound_sockets=127.19.246.193
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35109
--builtin_ntp_servers=127.19.246.212:34419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:24:32.132021 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:32.132340 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:32.132493 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:32.140673 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:32.140805 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:32.140981 9934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.193
I20260810 03:24:32.150945 9934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.246.193
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.193
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:32.152406 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:32.154295 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:32.168516 9940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:32.168704 9941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:32.169162 9943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:32.170033 9934 server_base.cc:1034] running on GCE node
I20260810 03:24:32.170732 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:32.172008 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:32.173280 9934 hybrid_clock.cc:648] HybridClock initialized: now 1786332272173220 us; error 56 us; skew 500 ppm
I20260810 03:24:32.173638 9934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:32.176414 9934 webserver.cc:460] Webserver started at http://127.19.246.193:36269/ using document root <none> and password file <none>
I20260810 03:24:32.177178 9934 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:32.177356 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:32.177881 9934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:32.182349 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/data/instance:
uuid: "191f7134f2eb4dbdb4edf1a2318ff034"
format_stamp: "Formatted at 2026-08-10 03:24:32 on dist-test-slave-btw8"
server_key: "3d78c3e5d7caa7e2b2214919e7d9f316"
server_key_iv: "50aba8d41aee3c41e4262034c0775f9d"
server_key_version: "encryptionkey@0"
I20260810 03:24:32.183228 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance:
uuid: "191f7134f2eb4dbdb4edf1a2318ff034"
format_stamp: "Formatted at 2026-08-10 03:24:32 on dist-test-slave-btw8"
server_key: "3d78c3e5d7caa7e2b2214919e7d9f316"
server_key_iv: "50aba8d41aee3c41e4262034c0775f9d"
server_key_version: "encryptionkey@0"
I20260810 03:24:32.189467 9934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:24:32.193611 9949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:32.195199 9934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:24:32.195452 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/wal
uuid: "191f7134f2eb4dbdb4edf1a2318ff034"
format_stamp: "Formatted at 2026-08-10 03:24:32 on dist-test-slave-btw8"
server_key: "3d78c3e5d7caa7e2b2214919e7d9f316"
server_key_iv: "50aba8d41aee3c41e4262034c0775f9d"
server_key_version: "encryptionkey@0"
I20260810 03:24:32.195714 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:32.214675 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:32.215843 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:32.216209 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:32.217299 9934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:32.218937 9934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:32.219058 9934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:32.219204 9934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:32.219307 9934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:32.255666 9934 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.193:38161
I20260810 03:24:32.255779 10080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.193:38161 every 8 connection(s)
I20260810 03:24:32.257105 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/data/info.pb
I20260810 03:24:32.267620 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 9934
I20260810 03:24:32.267850 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-0/wal/instance
I20260810 03:24:32.268304 20443 external_mini_cluster.cc:1428] Setting key 1752e9cffde08dc8980b6333cdf3d93c
I20260810 03:24:32.270800 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:32.259904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.193:55451 (local address 127.19.246.254:35109)
0810 03:24:32.260396 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:32.260401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:32.261653 (+ 1252us) server_negotiation.cc:407] Connection header received
0810 03:24:32.263583 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:32.263589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:32.263699 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:32.263829 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:32.264974 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:32.265737 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:32.267435 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:32.267983 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:32.268137 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:32.269894 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:32.269900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:32.269912 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:32.269932 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:32.269974 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:32.269986 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:32.270095 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:32.270349 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 03:24:32.270453 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":261,"threads_started":1}
I20260810 03:24:32.271164 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.194:0
--local_ip_for_outbound_sockets=127.19.246.194
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35109
--builtin_ntp_servers=127.19.246.212:34419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:24:32.272195 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:32.260501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35109 (local address 127.19.246.193:55451)
0810 03:24:32.261460 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:32.261504 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:32.263260 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:32.264079 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:32.264088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:32.264186 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:32.264781 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:32.264799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:32.265915 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:32.265925 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:32.267242 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:32.267255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:32.268250 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:32.268258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:32.268558 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:32.269501 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:32.269504 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:32.269579 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:32.269671 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:32.270116 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:32.270130 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:32.270142 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:24:32.270334 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 03:24:32.270802 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":297,"threads_started":1}
I20260810 03:24:32.273860 10081 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35109
I20260810 03:24:32.274201 10081 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:32.275204 10081 heartbeater.cc:511] Master 127.19.246.254:35109 requested a full tablet report, sending...
I20260810 03:24:32.277791 9865 ts_manager.cc:194] Registered new tserver with Master: 191f7134f2eb4dbdb4edf1a2318ff034 (127.19.246.193:38161)
I20260810 03:24:32.280045 9865 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.193:55451
W20260810 03:24:32.635655 10086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:32.635967 10086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:32.636121 10086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:32.644028 10086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:32.644150 10086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:32.644320 10086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.194
I20260810 03:24:32.654328 10086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.246.194
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.194
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:32.655728 10086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:32.657541 10086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:32.671787 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:32.671814 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:32.671734 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:32.673111 10086 server_base.cc:1034] running on GCE node
I20260810 03:24:32.673935 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:32.675088 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:32.676370 10086 hybrid_clock.cc:648] HybridClock initialized: now 1786332272676292 us; error 66 us; skew 500 ppm
I20260810 03:24:32.676800 10086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:32.679524 10086 webserver.cc:460] Webserver started at http://127.19.246.194:36743/ using document root <none> and password file <none>
I20260810 03:24:32.680317 10086 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:32.680415 10086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:32.680800 10086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:32.684011 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/data/instance:
uuid: "db5330059e324deabe8bffeb22f91588"
format_stamp: "Formatted at 2026-08-10 03:24:32 on dist-test-slave-btw8"
server_key: "344b48e559feb27da59e3aa90cdeb057"
server_key_iv: "a66f30941f39770ea2c51003ec909ba8"
server_key_version: "encryptionkey@0"
I20260810 03:24:32.685009 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance:
uuid: "db5330059e324deabe8bffeb22f91588"
format_stamp: "Formatted at 2026-08-10 03:24:32 on dist-test-slave-btw8"
server_key: "344b48e559feb27da59e3aa90cdeb057"
server_key_iv: "a66f30941f39770ea2c51003ec909ba8"
server_key_version: "encryptionkey@0"
I20260810 03:24:32.690991 10086 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:24:32.695021 10102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:32.696560 10086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:24:32.696781 10086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/wal
uuid: "db5330059e324deabe8bffeb22f91588"
format_stamp: "Formatted at 2026-08-10 03:24:32 on dist-test-slave-btw8"
server_key: "344b48e559feb27da59e3aa90cdeb057"
server_key_iv: "a66f30941f39770ea2c51003ec909ba8"
server_key_version: "encryptionkey@0"
I20260810 03:24:32.697027 10086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:32.717272 10086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:32.718276 10086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:32.718622 10086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:32.719775 10086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:32.721511 10086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:32.721609 10086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:32.721781 10086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:32.721873 10086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:32.761862 10086 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.194:35721
I20260810 03:24:32.761962 10232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.194:35721 every 8 connection(s)
I20260810 03:24:32.763334 10086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/data/info.pb
I20260810 03:24:32.769819 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 10086
I20260810 03:24:32.770022 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/wal/instance
I20260810 03:24:32.770649 20443 external_mini_cluster.cc:1428] Setting key 1e6162cf73d498578fb4108326f49a7d
I20260810 03:24:32.774078 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.246.195:0
--local_ip_for_outbound_sockets=127.19.246.195
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35109
--builtin_ntp_servers=127.19.246.212:34419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:24:32.778625 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:32.766264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.194:54761 (local address 127.19.246.254:35109)
0810 03:24:32.766513 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:32.766521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:32.768188 (+ 1667us) server_negotiation.cc:407] Connection header received
0810 03:24:32.770253 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:32.770258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:32.770340 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:32.770445 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:32.771833 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:32.773165 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:32.775002 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:32.775826 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:32.775983 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:32.777574 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:32.777583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:32.777591 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:32.777620 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:32.777666 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:32.777687 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:32.777787 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:32.778064 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 03:24:32.778189 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:24:32.779824 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:32.766908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35109 (local address 127.19.246.194:54761)
0810 03:24:32.768002 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:32.768048 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:32.769915 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:32.770741 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:32.770754 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:32.770874 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:32.771662 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:32.771687 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:32.773379 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:32.773388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:32.774841 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:32.774854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:32.776090 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:32.776096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:32.776351 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:32.777206 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:32.777209 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:32.777256 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:32.777353 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:32.777826 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:32.777835 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:32.777842 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:24:32.778030 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 03:24:32.778415 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":381,"threads_started":1}
I20260810 03:24:32.781479 10233 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35109
I20260810 03:24:32.781872 10233 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:32.782735 10233 heartbeater.cc:511] Master 127.19.246.254:35109 requested a full tablet report, sending...
I20260810 03:24:32.784353 9865 ts_manager.cc:194] Registered new tserver with Master: db5330059e324deabe8bffeb22f91588 (127.19.246.194:35721)
I20260810 03:24:32.785393 9865 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.194:54761
W20260810 03:24:33.141661 10236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:33.141980 10236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:33.142172 10236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:33.151189 10236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:33.151336 10236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:33.151556 10236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.246.195
I20260810 03:24:33.161430 10236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:34419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.246.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.246.195
--webserver_port=0
--tserver_master_addrs=127.19.246.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.246.195
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:33.162858 10236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:33.164795 10236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:33.180287 10245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:33.180310 10241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:33.180351 10243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:33.182700 10236 server_base.cc:1034] running on GCE node
I20260810 03:24:33.184077 10236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:33.185632 10236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:33.186962 10236 hybrid_clock.cc:648] HybridClock initialized: now 1786332273186873 us; error 87 us; skew 500 ppm
I20260810 03:24:33.187520 10236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 03:24:33.190680 10236 webserver.cc:460] Webserver started at http://127.19.246.195:40721/ using document root <none> and password file <none>
I20260810 03:24:33.191707 10236 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:33.191877 10236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:33.192262 10236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:33.196110 10236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/data/instance:
uuid: "2a3739e8863641e39b0c2c04a2c2727c"
format_stamp: "Formatted at 2026-08-10 03:24:33 on dist-test-slave-btw8"
server_key: "ef24ad979d678c76eaf708b4236aefd6"
server_key_iv: "ef557dd3901562b1720cc7a0a5c9c4d7"
server_key_version: "encryptionkey@0"
I20260810 03:24:33.197114 10236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance:
uuid: "2a3739e8863641e39b0c2c04a2c2727c"
format_stamp: "Formatted at 2026-08-10 03:24:33 on dist-test-slave-btw8"
server_key: "ef24ad979d678c76eaf708b4236aefd6"
server_key_iv: "ef557dd3901562b1720cc7a0a5c9c4d7"
server_key_version: "encryptionkey@0"
I20260810 03:24:33.204191 10236 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260810 03:24:33.208793 10251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:33.210574 10236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:24:33.210873 10236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/wal
uuid: "2a3739e8863641e39b0c2c04a2c2727c"
format_stamp: "Formatted at 2026-08-10 03:24:33 on dist-test-slave-btw8"
server_key: "ef24ad979d678c76eaf708b4236aefd6"
server_key_iv: "ef557dd3901562b1720cc7a0a5c9c4d7"
server_key_version: "encryptionkey@0"
I20260810 03:24:33.211135 10236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:33.235335 10236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:33.236312 10236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:33.236686 10236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:33.237800 10236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:24:33.239583 10236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:24:33.239759 10236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:33.239897 10236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:24:33.239990 10236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:33.277449 10236 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.195:45197
I20260810 03:24:33.277549 10376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.195:45197 every 8 connection(s)
I20260810 03:24:33.278971 10236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/data/info.pb
I20260810 03:24:33.283769 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 10236
I20260810 03:24:33.283996 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/wal/instance
I20260810 03:24:33.284329 10081 heartbeater.cc:503] Master 127.19.246.254:35109 was elected leader, sending a full tablet report...
I20260810 03:24:33.284694 20443 external_mini_cluster.cc:1428] Setting key c50e87bdb74da65cc0dd229e0940c5fc
I20260810 03:24:33.295503 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.282082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:39551 (local address 127.19.246.254:35109)
0810 03:24:33.282591 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.282597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.284428 (+ 1831us) server_negotiation.cc:407] Connection header received
0810 03:24:33.287340 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.287347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.287499 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.287664 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:33.289082 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.290038 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.292041 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.292554 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.292712 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.294414 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:33.294420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:33.294424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:33.294447 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:33.294492 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:33.294506 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:33.294620 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.294878 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.294996 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
I20260810 03:24:33.296656 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.282756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:35109 (local address 127.19.246.195:39551)
0810 03:24:33.284210 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.284274 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:33.287018 (+ 2744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:33.287945 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:33.287958 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:33.288149 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:24:33.288877 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.288898 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.290246 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.290250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.291851 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.291866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.292846 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.292854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.293128 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.293985 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:24:33.293988 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:24:33.294039 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:24:33.294142 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:24:33.294620 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:24:33.294636 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:24:33.294648 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 03:24:33.294867 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 03:24:33.295397 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":433,"threads_started":1}
I20260810 03:24:33.298466 10377 heartbeater.cc:347] Connected to a master server at 127.19.246.254:35109
I20260810 03:24:33.298871 10377 heartbeater.cc:464] Registering TS with master...
I20260810 03:24:33.300161 10377 heartbeater.cc:511] Master 127.19.246.254:35109 requested a full tablet report, sending...
I20260810 03:24:33.302062 9864 ts_manager.cc:194] Registered new tserver with Master: 2a3739e8863641e39b0c2c04a2c2727c (127.19.246.195:45197)
I20260810 03:24:33.303193 9864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.246.195:39551
I20260810 03:24:33.314458 20443 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:24:33.325848 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.319311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.19.246.254:35109)
0810 03:24:33.319593 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.319601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.319960 (+ 359us) server_negotiation.cc:407] Connection header received
0810 03:24:33.320199 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.320205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.320285 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.320401 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:33.320992 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.321801 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.323279 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.323733 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.323881 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.325088 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:33.325092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:33.325095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:33.325122 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:33.325157 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:33.325169 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:33.325271 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.325418 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.325522 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 03:24:33.330828 9864 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:24:33.339201 9864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:24:33.382663 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.356688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47804)
0810 03:24:33.372166 (+ 15478us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.372188 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:33.372427 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:33.373681 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:33.373689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:33.373723 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:33.374297 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.374315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.376624 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.376631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.379646 (+ 3015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.379663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.381672 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.381676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.381834 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.381866 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:24:33.382008 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 03:24:33.382233 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15347,"mutex_wait_us":217,"thread_start_us":14231,"threads_started":1}
I20260810 03:24:33.383062 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.354008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.195:45197 (local address 127.0.0.1:33276)
0810 03:24:33.369154 (+ 15146us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.369229 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:33.369575 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:33.374096 (+ 4521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:33.374100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:33.374130 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:33.374644 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.374655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.376496 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.376503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.380043 (+ 3540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.380058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.382097 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.382103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.382263 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.382290 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:24:33.382381 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 03:24:33.382520 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14704,"mutex_wait_us":168,"thread_start_us":15834,"threads_started":1}
I20260810 03:24:33.383684 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.356863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.19.246.193:38161)
0810 03:24:33.364231 (+ 7368us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.364246 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.372303 (+ 8057us) server_negotiation.cc:407] Connection header received
0810 03:24:33.372654 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.372663 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.373026 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.373258 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:33.374564 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.376478 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.379901 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.381187 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.381329 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.381498 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.382986 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.383190 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7143,"thread_start_us":6976,"threads_started":1}
I20260810 03:24:33.384514 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.354421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.19.246.195:45197)
0810 03:24:33.373462 (+ 19041us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.373471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.373508 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:24:33.373626 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.373634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.373871 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.374095 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:33.374830 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.376372 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.380278 (+ 3906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.381881 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.382102 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.382304 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.382426 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.382584 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18829,"thread_start_us":18616,"threads_started":1}
I20260810 03:24:33.385913 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.354427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.19.246.194:35721)
0810 03:24:33.373390 (+ 18963us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.373401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.373433 (+ 32us) server_negotiation.cc:407] Connection header received
0810 03:24:33.373539 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.373546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.373730 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.374320 (+ 590us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:33.375764 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.377156 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.380144 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.381855 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.382071 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.382304 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.383687 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.385524 (+ 1837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17922,"thread_start_us":18169,"threads_started":1}
I20260810 03:24:33.386312 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.354147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:35721 (local address 127.0.0.1:43352)
0810 03:24:33.371371 (+ 17224us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.371401 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:33.371576 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:33.374438 (+ 2862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:33.374446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:33.374985 (+ 539us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:33.375507 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.375523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.377323 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.377329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.379806 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.379825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.382787 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.382800 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.382979 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.383028 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 03:24:33.383111 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 03:24:33.383243 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16980,"mutex_wait_us":103,"thread_start_us":17739,"threads_started":1}
I20260810 03:24:33.388628 10003 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 (DEFAULT_TABLE table=test-table [id=42288c9c7e4b402ea88dae7f2e0b4d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:33.388631 10308 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 (DEFAULT_TABLE table=test-table [id=42288c9c7e4b402ea88dae7f2e0b4d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:33.389758 10159 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 (DEFAULT_TABLE table=test-table [id=42288c9c7e4b402ea88dae7f2e0b4d97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:24:33.390722 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:33.391165 10308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:33.391197 10159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:33.408185 10402 tablet_bootstrap.cc:492] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034: Bootstrap starting.
I20260810 03:24:33.411126 10400 tablet_bootstrap.cc:492] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Bootstrap starting.
I20260810 03:24:33.412180 10401 tablet_bootstrap.cc:492] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Bootstrap starting.
I20260810 03:24:33.415019 10400 tablet_bootstrap.cc:654] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:33.415627 10401 tablet_bootstrap.cc:654] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:33.416566 10400 log.cc:824] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:33.416769 10401 log.cc:824] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:33.418701 10402 tablet_bootstrap.cc:654] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:33.419867 10400 tablet_bootstrap.cc:492] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: No bootstrap required, opened a new log
I20260810 03:24:33.419852 10402 log.cc:824] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:33.420182 10400 ts_tablet_manager.cc:1397] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20260810 03:24:33.423308 10402 tablet_bootstrap.cc:492] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034: No bootstrap required, opened a new log
I20260810 03:24:33.423310 10401 tablet_bootstrap.cc:492] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: No bootstrap required, opened a new log
I20260810 03:24:33.423681 10401 ts_tablet_manager.cc:1397] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260810 03:24:33.423693 10402 ts_tablet_manager.cc:1397] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260810 03:24:33.429090 10400 raft_consensus.cc:348] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.429556 10400 raft_consensus.cc:374] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:33.429664 10400 raft_consensus.cc:729] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5330059e324deabe8bffeb22f91588, State: Initialized, Role: FOLLOWER
I20260810 03:24:33.430361 10400 consensus_queue.cc:260] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.431841 10233 heartbeater.cc:503] Master 127.19.246.254:35109 was elected leader, sending a full tablet report...
I20260810 03:24:33.432437 10400 ts_tablet_manager.cc:1428] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260810 03:24:33.434773 10401 raft_consensus.cc:348] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.435213 10401 raft_consensus.cc:374] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:33.435338 10401 raft_consensus.cc:729] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3739e8863641e39b0c2c04a2c2727c, State: Initialized, Role: FOLLOWER
I20260810 03:24:33.436028 10401 consensus_queue.cc:260] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.435962 10402 raft_consensus.cc:348] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.436461 10402 raft_consensus.cc:374] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:33.436570 10402 raft_consensus.cc:729] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191f7134f2eb4dbdb4edf1a2318ff034, State: Initialized, Role: FOLLOWER
I20260810 03:24:33.437158 10402 consensus_queue.cc:260] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.440014 10402 ts_tablet_manager.cc:1428] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260810 03:24:33.441411 10377 heartbeater.cc:503] Master 127.19.246.254:35109 was elected leader, sending a full tablet report...
I20260810 03:24:33.442054 10401 ts_tablet_manager.cc:1428] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260810 03:24:33.509935 10083 tablet.cc:2369] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:24:33.515802 10234 tablet.cc:2369] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:24:33.531095 10379 tablet.cc:2369] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:24:33.536830 10407 raft_consensus.cc:482] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:24:33.537177 10407 raft_consensus.cc:504] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.539067 10407 leader_election.cc:288] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db5330059e324deabe8bffeb22f91588 (127.19.246.194:35721), 191f7134f2eb4dbdb4edf1a2318ff034 (127.19.246.193:38161)
I20260810 03:24:33.546445 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.539860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:40469 (local address 127.19.246.194:35721)
0810 03:24:33.540106 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.540113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.540136 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:24:33.540316 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.540322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.540405 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.540524 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:33.541075 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.542178 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.544373 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.545538 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.545676 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.545732 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.546118 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.546221 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 03:24:33.546562 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.539521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.194:35721 (local address 127.19.246.195:40469)
0810 03:24:33.539912 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.539937 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:33.540109 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:33.540565 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:33.540570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:33.540602 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:33.540874 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.540888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.542345 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.542352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.544169 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.544184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.545724 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.545733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.545917 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.545958 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:24:33.546071 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 03:24:33.546249 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260810 03:24:33.547724 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.539592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.19.246.195:58305)
0810 03:24:33.540661 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.540677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:24:33.540798 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:24:33.541202 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:24:33.541210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:24:33.541239 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:24:33.541698 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.541710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.543261 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.543265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.545050 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:24:33.545067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.546954 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.546963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:24:33.547163 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.547209 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 03:24:33.547311 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 03:24:33.547493 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":675,"threads_started":1}
I20260810 03:24:33.547874 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.539799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.246.195:58305 (local address 127.19.246.193:38161)
0810 03:24:33.540055 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.540065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.540778 (+ 713us) server_negotiation.cc:407] Connection header received
0810 03:24:33.540910 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.540917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.541030 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.541123 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:24:33.541933 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.543048 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.545337 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.546540 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.546835 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.546921 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.547393 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.547559 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 03:24:33.547955 10182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1dc0c3ffda4abc9d6e6c7c5de3fad9" candidate_uuid: "2a3739e8863641e39b0c2c04a2c2727c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db5330059e324deabe8bffeb22f91588" is_pre_election: true
I20260810 03:24:33.548645 10182 raft_consensus.cc:2454] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3739e8863641e39b0c2c04a2c2727c in term 0.
I20260810 03:24:33.548936 10030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1dc0c3ffda4abc9d6e6c7c5de3fad9" candidate_uuid: "2a3739e8863641e39b0c2c04a2c2727c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" is_pre_election: true
I20260810 03:24:33.549492 10257 leader_election.cc:302] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3739e8863641e39b0c2c04a2c2727c, db5330059e324deabe8bffeb22f91588; no voters:
I20260810 03:24:33.549611 10030 raft_consensus.cc:2454] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3739e8863641e39b0c2c04a2c2727c in term 0.
I20260810 03:24:33.550020 10407 raft_consensus.cc:2781] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:24:33.550204 10407 raft_consensus.cc:482] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:24:33.550326 10407 raft_consensus.cc:3037] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:33.552845 10407 raft_consensus.cc:504] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.553516 10407 leader_election.cc:288] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [CANDIDATE]: Term 1 election: Requested vote from peers db5330059e324deabe8bffeb22f91588 (127.19.246.194:35721), 191f7134f2eb4dbdb4edf1a2318ff034 (127.19.246.193:38161)
I20260810 03:24:33.554155 10030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1dc0c3ffda4abc9d6e6c7c5de3fad9" candidate_uuid: "2a3739e8863641e39b0c2c04a2c2727c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191f7134f2eb4dbdb4edf1a2318ff034"
I20260810 03:24:33.554246 10182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1dc0c3ffda4abc9d6e6c7c5de3fad9" candidate_uuid: "2a3739e8863641e39b0c2c04a2c2727c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db5330059e324deabe8bffeb22f91588"
I20260810 03:24:33.554435 10030 raft_consensus.cc:3037] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:33.554549 10182 raft_consensus.cc:3037] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:33.557163 10030 raft_consensus.cc:2454] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3739e8863641e39b0c2c04a2c2727c in term 1.
I20260810 03:24:33.557173 10182 raft_consensus.cc:2454] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3739e8863641e39b0c2c04a2c2727c in term 1.
I20260810 03:24:33.557893 10253 leader_election.cc:302] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 191f7134f2eb4dbdb4edf1a2318ff034, 2a3739e8863641e39b0c2c04a2c2727c, db5330059e324deabe8bffeb22f91588; no voters:
I20260810 03:24:33.558516 10407 raft_consensus.cc:2781] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:33.559087 10407 raft_consensus.cc:686] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 1 LEADER]: Becoming Leader. State: Replica: 2a3739e8863641e39b0c2c04a2c2727c, State: Running, Role: LEADER
I20260810 03:24:33.559708 10407 consensus_queue.cc:237] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } }
I20260810 03:24:33.564937 9863 catalog_manager.cc:5495] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a3739e8863641e39b0c2c04a2c2727c (127.19.246.195). New cstate: current_term: 1 leader_uuid: "2a3739e8863641e39b0c2c04a2c2727c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3739e8863641e39b0c2c04a2c2727c" member_type: VOTER last_known_addr { host: "127.19.246.195" port: 45197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 } health_report { overall_health: UNKNOWN } } }
I20260810 03:24:33.583268 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.577164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.19.246.195:45197)
0810 03:24:33.577492 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.577498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.577649 (+ 151us) server_negotiation.cc:407] Connection header received
0810 03:24:33.577923 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.577927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.578024 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.578199 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:24:33.578623 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.579445 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.580868 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.581321 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.581462 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.582022 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:24:33.582582 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:24:33.582771 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.582888 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.582990 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260810 03:24:33.592839 10030 raft_consensus.cc:1261] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034 [term 1 FOLLOWER]: Refusing update from remote peer 2a3739e8863641e39b0c2c04a2c2727c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:33.592839 10182 raft_consensus.cc:1261] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 1 FOLLOWER]: Refusing update from remote peer 2a3739e8863641e39b0c2c04a2c2727c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:24:33.593993 10411 consensus_queue.cc:1035] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [LEADER]: Connected to new peer: Peer: permanent_uuid: "191f7134f2eb4dbdb4edf1a2318ff034" member_type: VOTER last_known_addr { host: "127.19.246.193" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:33.594430 10407 consensus_queue.cc:1035] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [LEADER]: Connected to new peer: Peer: permanent_uuid: "db5330059e324deabe8bffeb22f91588" member_type: VOTER last_known_addr { host: "127.19.246.194" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:24:33.608405 10414 mvcc.cc:205] Tried to move back new op lower bound from 7316816992625020928 to 7316816992505131008. Current Snapshot: MvccSnapshot[applied={T|T < 7316816992625020928}]
I20260810 03:24:33.631057 9863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:33.631477 9863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:24:33.635751 9863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31c25f1b5fdf4897a0fab21093a8081e: Sending DeleteTablet for 3 replicas of tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9
I20260810 03:24:33.637269 10308 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:33 UTC) from {username='slave'} at 127.0.0.1:33276
I20260810 03:24:33.637267 10159 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:33 UTC) from {username='slave'} at 127.0.0.1:43352
I20260810 03:24:33.637400 10003 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:24:33 UTC) from {username='slave'} at 127.0.0.1:47804
I20260810 03:24:33.639179 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 9934
I20260810 03:24:33.664513 10425 tablet_replica.cc:330] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: stopping tablet replica
I20260810 03:24:33.665057 10425 raft_consensus.cc:2229] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:24:33.665669 10425 raft_consensus.cc:2258] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:33.669147 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.648191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.19.246.193:38161)
0810 03:24:33.648390 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.648397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.657146 (+ 8749us) server_negotiation.cc:407] Connection header received
0810 03:24:33.657732 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.657739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.657866 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.657960 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:33.658698 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.660212 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.662069 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.662802 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.662895 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.666647 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:33.666663 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:33.666684 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:33.667623 (+ 939us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:33.667699 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:33.667732 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:33.667863 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.668566 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.668726 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:24:33.670603 10075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:33.672668 10425 ts_tablet_manager.cc:1905] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:33.681414 10424 tablet_replica.cc:330] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: stopping tablet replica
I20260810 03:24:33.681704 10425 ts_tablet_manager.cc:1918] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:33.682083 10425 log.cc:1196] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-1/wal/wals/8b1dc0c3ffda4abc9d6e6c7c5de3fad9
I20260810 03:24:33.681998 10424 raft_consensus.cc:2229] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:24:33.682801 10424 raft_consensus.cc:2258] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:24:33.683003 10425 ts_tablet_manager.cc:1939] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P db5330059e324deabe8bffeb22f91588: Deleting consensus metadata
I20260810 03:24:33.685184 10424 ts_tablet_manager.cc:1905] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:24:33.686417 9849 catalog_manager.cc:4841] TS db5330059e324deabe8bffeb22f91588 (127.19.246.194:35721): tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 (table test-table [id=42288c9c7e4b402ea88dae7f2e0b4d97]) successfully deleted
I20260810 03:24:33.692790 10424 ts_tablet_manager.cc:1918] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:24:33.693044 10424 log.cc:1196] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Deleting WAL directory at /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786331905428759-20443-0/minicluster-data/ts-2/wal/wals/8b1dc0c3ffda4abc9d6e6c7c5de3fad9
I20260810 03:24:33.693678 10424 ts_tablet_manager.cc:1939] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 2a3739e8863641e39b0c2c04a2c2727c: Deleting consensus metadata
I20260810 03:24:33.695629 9846 catalog_manager.cc:4841] TS 2a3739e8863641e39b0c2c04a2c2727c (127.19.246.195:45197): tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 (table test-table [id=42288c9c7e4b402ea88dae7f2e0b4d97]) successfully deleted
I20260810 03:24:33.811839 10426 tablet_replica.cc:330] T 8b1dc0c3ffda4abc9d6e6c7c5de3fad9 P 191f7134f2eb4dbdb4edf1a2318ff034: stopping tablet replica
I20260810 03:24:33.812049 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 9934
W20260810 03:24:33.839653 9847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.246.193:38161 (error 108)
I20260810 03:24:33.839996 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10086
I20260810 03:24:33.842008 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.841495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47810)
0810 03:24:33.841651 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.841816 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:24:33.842327 9847 catalog_manager.cc:4584] TS 191f7134f2eb4dbdb4edf1a2318ff034 (127.19.246.193:38161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b1dc0c3ffda4abc9d6e6c7c5de3fad9: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
I20260810 03:24:33.849417 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.841178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.19.246.194:35721)
0810 03:24:33.841431 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.841438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.841466 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:24:33.841544 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.841550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.841667 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.841808 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:33.842418 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.843325 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.845543 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.846026 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.846165 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.847783 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:33.847792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:33.847808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:33.848495 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:33.848550 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:33.848573 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:33.848670 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.848879 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.848997 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:24:33.850284 10228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:33.896147 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.895565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47812)
0810 03:24:33.895812 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.895947 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 03:24:33.897593 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.897086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47826)
0810 03:24:33.897291 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.897398 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 03:24:33.950273 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 10086
I20260810 03:24:33.958722 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.958001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47840)
0810 03:24:33.958340 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.958472 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260810 03:24:33.960289 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.959794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47848)
0810 03:24:33.960002 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:33.960104 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 03:24:33.974083 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10236
I20260810 03:24:33.984935 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:33.975011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.19.246.195:45197)
0810 03:24:33.975208 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:33.975213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:33.975585 (+ 372us) server_negotiation.cc:407] Connection header received
0810 03:24:33.975869 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:33.975878 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:33.976098 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:33.976287 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:33.977406 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.979029 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.981056 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:33.981593 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:33.981867 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:33.983189 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:33.983199 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:33.983215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:33.983985 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:33.984043 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:33.984075 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:33.984209 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:33.984371 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 03:24:33.984499 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:24:33.986340 10372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:34.041363 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:34.040717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47862)
0810 03:24:34.041004 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:34.041127 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 03:24:34.042817 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:34.042250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47868)
0810 03:24:34.042507 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:34.042660 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260810 03:24:34.120365 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 10236
I20260810 03:24:34.149302 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 9828
I20260810 03:24:34.150622 9903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:34.271826 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:34.271255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.193:38161 (local address 127.0.0.1:47884)
0810 03:24:34.271503 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 03:24:34.271626 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.193:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 03:24:34.271924 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 9828
2026-08-10T03:24:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3234 ms)
[----------] 6 tests from EncryptionPolicyTest (25414 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:34 dist-test-slave-btw8 krb5kdc[10436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:34 dist-test-slave-btw8 krb5kdc[10436](info): set up 2 sockets
Aug 10 03:24:34 dist-test-slave-btw8 krb5kdc[10436](info): commencing operation
krb5kdc: starting...
W20260810 03:24:35.524885 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:35 dist-test-slave-btw8 krb5kdc[10436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:35.907879 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:36.282042 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:36.282413 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:36.282513 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:36.290500 10450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:36.290659 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:36.290710 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:36.290768 10450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:36.290817 10450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:36.301541 10450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:36.302973 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:36.304512 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:36.318117 10459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:36.318028 10457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:36.318064 10456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:36.319468 10450 server_base.cc:1034] running on GCE node
I20260810 03:24:36.320429 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:36.322171 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:36.323464 10450 hybrid_clock.cc:648] HybridClock initialized: now 1786332276323394 us; error 61 us; skew 500 ppm
I20260810 03:24:36.323818 10450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:36 dist-test-slave-btw8 krb5kdc[10436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:36.327309 10450 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:36.329157 10450 webserver.cc:460] Webserver started at http://127.0.0.1:40825/ using document root <none> and password file <none>
I20260810 03:24:36.329905 10450 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:36.330020 10450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:36.330297 10450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:36.333791 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "942a0e07a3a0474db50ab583a5837664"
format_stamp: "Formatted at 2026-08-10 03:24:36 on dist-test-slave-btw8"
server_key: "26603269bbd7261820c6689b01ee2732"
server_key_iv: "fad140f255906a4ddd0bd8c04075b406"
server_key_version: "encryptionkey@0"
I20260810 03:24:36.334707 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "942a0e07a3a0474db50ab583a5837664"
format_stamp: "Formatted at 2026-08-10 03:24:36 on dist-test-slave-btw8"
server_key: "26603269bbd7261820c6689b01ee2732"
server_key_iv: "fad140f255906a4ddd0bd8c04075b406"
server_key_version: "encryptionkey@0"
I20260810 03:24:36.340757 10450 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260810 03:24:36.345054 10467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:36.346841 10450 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:24:36.347101 10450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "942a0e07a3a0474db50ab583a5837664"
format_stamp: "Formatted at 2026-08-10 03:24:36 on dist-test-slave-btw8"
server_key: "26603269bbd7261820c6689b01ee2732"
server_key_iv: "fad140f255906a4ddd0bd8c04075b406"
server_key_version: "encryptionkey@0"
I20260810 03:24:36.347342 10450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:36.390443 10450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:36.396095 10450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:36.396441 10450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:36.418390 10450 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33287
I20260810 03:24:36.418450 10528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33287 every 8 connection(s)
I20260810 03:24:36.420147 10450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:36.424563 10529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:36.427127 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 10450
I20260810 03:24:36.427407 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:36.428260 20443 external_mini_cluster.cc:1428] Setting key 0c4a184391fd0c320aec42b12bc40d18
I20260810 03:24:36.438872 10529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664: Bootstrap starting.
I20260810 03:24:36.443240 10529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:36.444742 10529 log.cc:824] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:24:36 dist-test-slave-btw8 krb5kdc[10436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:24:36.448261 10529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664: No bootstrap required, opened a new log
I20260810 03:24:36.456365 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:36.430560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54953 (local address 127.0.0.1:33287)
0810 03:24:36.431583 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:36.431596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:36.431650 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:24:36.433513 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:36.433552 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:36.434012 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:36.434397 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:36.436624 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:36.438122 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:36.440089 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:36.440933 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:36.441119 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:36.446760 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:36.446770 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:36.446785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:36.446831 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:36.450476 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:36.451708 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:36.451722 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:36.451732 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:36.451977 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:36.452617 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:36.452622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:36.452626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:36.453404 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:36.453812 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:36.454331 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 03:24:36.454743 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":451,"threads_started":1}
I20260810 03:24:36.459331 10529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942a0e07a3a0474db50ab583a5837664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } }
I20260810 03:24:36.459865 10529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:36.459997 10529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 942a0e07a3a0474db50ab583a5837664, State: Initialized, Role: FOLLOWER
I20260810 03:24:36.460698 10529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942a0e07a3a0474db50ab583a5837664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } }
I20260810 03:24:36.460978 10529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:36.461162 10529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:36.461351 10529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:36.461539 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:36.464006 10529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942a0e07a3a0474db50ab583a5837664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } }
I20260810 03:24:36.464604 10529 leader_election.cc:302] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 942a0e07a3a0474db50ab583a5837664; no voters:
I20260810 03:24:36.465399 10529 leader_election.cc:288] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:36.465531 10538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:36.466794 10538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [term 1 LEADER]: Becoming Leader. State: Replica: 942a0e07a3a0474db50ab583a5837664, State: Running, Role: LEADER
I20260810 03:24:36.468024 10538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942a0e07a3a0474db50ab583a5837664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } }
I20260810 03:24:36.468210 10529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:36.473909 10541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 942a0e07a3a0474db50ab583a5837664. Latest consensus state: current_term: 1 leader_uuid: "942a0e07a3a0474db50ab583a5837664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942a0e07a3a0474db50ab583a5837664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } } }
I20260810 03:24:36.474480 10541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:36.474807 10540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "942a0e07a3a0474db50ab583a5837664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "942a0e07a3a0474db50ab583a5837664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33287 } } }
I20260810 03:24:36.475162 10540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:36.480602 10548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:36.487179 10548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:36.490813 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:36.467620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54693 (local address 127.0.0.1:33287)
0810 03:24:36.467819 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:36.467826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:36.468195 (+ 369us) server_negotiation.cc:407] Connection header received
0810 03:24:36.468444 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:36.468449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:36.468527 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:36.468587 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:36.470309 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:36.471122 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:36.473422 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:36.474393 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:36.474516 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:36.478498 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:36.478509 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:36.478517 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:36.478597 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:36.484040 (+ 5443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:36.485780 (+ 1740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:36.485790 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:36.485797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:36.486035 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:36.487070 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:36.487079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:36.487085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:36.487727 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:36.488116 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:36.489876 (+ 1760us) server_negotiation.cc:299] Negotiation successful
0810 03:24:36.490169 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:24:36.495893 10558 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:24:36.498679 10548 catalog_manager.cc:1347] Generated new cluster ID: 864ba38438a8477ea1a8ef85a5dca821
I20260810 03:24:36.498839 10548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:36.513183 10548 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:36.515502 10548 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:36.529287 10548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 942a0e07a3a0474db50ab583a5837664: Generated new TSK 0
I20260810 03:24:36.530445 10548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:36.547063 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10450
I20260810 03:24:36.548710 10523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:36.646862 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 10450
W20260810 03:24:36.669379 10433 connection.cc:535] client connection to 127.0.0.1:33287 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33287): Connection reset by peer (error 104)
2026-08-10T03:24:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:36 dist-test-slave-btw8 krb5kdc[10563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:36 dist-test-slave-btw8 krb5kdc[10563](info): set up 2 sockets
Aug 10 03:24:36 dist-test-slave-btw8 krb5kdc[10563](info): commencing operation
krb5kdc: starting...
W20260810 03:24:37.889096 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.005s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:38 dist-test-slave-btw8 krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:38.264284 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:38.650552 10577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:38.650898 10577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:38.650998 10577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:38.658854 10577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:38.659004 10577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:38.659058 10577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:38.659116 10577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:38.659165 10577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:38.669981 10577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:38.671569 10577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:38.672909 10577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:38.685810 10582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:38.685756 10583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:38.685729 10587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:38.686825 10577 server_base.cc:1034] running on GCE node
I20260810 03:24:38.687916 10577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:38.689571 10577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:38.690851 10577 hybrid_clock.cc:648] HybridClock initialized: now 1786332278690792 us; error 60 us; skew 500 ppm
I20260810 03:24:38.691293 10577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:38 dist-test-slave-btw8 krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:38.694983 10577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:38.696903 10577 webserver.cc:460] Webserver started at http://127.0.0.1:45211/ using document root <none> and password file <none>
I20260810 03:24:38.697692 10577 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:38.697863 10577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:38.698203 10577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:38.701723 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "37ff71860ab54680a1e729e1c2d81a5f"
format_stamp: "Formatted at 2026-08-10 03:24:38 on dist-test-slave-btw8"
server_key: "c8b6a9a2a699321a1531191d092f9562"
server_key_iv: "55d843fd958760d72b3335e827aebb85"
server_key_version: "encryptionkey@0"
I20260810 03:24:38.702598 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "37ff71860ab54680a1e729e1c2d81a5f"
format_stamp: "Formatted at 2026-08-10 03:24:38 on dist-test-slave-btw8"
server_key: "c8b6a9a2a699321a1531191d092f9562"
server_key_iv: "55d843fd958760d72b3335e827aebb85"
server_key_version: "encryptionkey@0"
I20260810 03:24:38.708642 10577 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 03:24:38.712719 10594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:38.714370 10577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:24:38.714618 10577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "37ff71860ab54680a1e729e1c2d81a5f"
format_stamp: "Formatted at 2026-08-10 03:24:38 on dist-test-slave-btw8"
server_key: "c8b6a9a2a699321a1531191d092f9562"
server_key_iv: "55d843fd958760d72b3335e827aebb85"
server_key_version: "encryptionkey@0"
I20260810 03:24:38.714861 10577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:38.727149 10577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:38.732355 10577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:38.732694 10577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:38.753593 10577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45111
I20260810 03:24:38.753625 10651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45111 every 8 connection(s)
I20260810 03:24:38.755213 10577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:38.759711 10652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:38.762871 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 10577
I20260810 03:24:38.763154 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:38.763725 20443 external_mini_cluster.cc:1428] Setting key e29c83888cb318303f1b33372305bf48
I20260810 03:24:38.772852 10652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f: Bootstrap starting.
I20260810 03:24:38.776705 10652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:38.777945 10652 log.cc:824] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:38.781404 10652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f: No bootstrap required, opened a new log
I20260810 03:24:38.784507 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:38.765924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43347 (local address 127.0.0.1:45111)
0810 03:24:38.766848 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:38.766861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:38.766906 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:24:38.768955 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:38.768982 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:38.769452 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:38.769905 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:38.770551 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:38.772057 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:38.773607 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:38.774313 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:38.774547 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:38.781724 (+ 7177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:38.781733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:38.781749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:38.782357 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:38.782442 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:38.782473 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:38.782770 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:38.783103 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 03:24:38.783294 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":437,"threads_started":1}
I20260810 03:24:38.789537 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:38.791119 10652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ff71860ab54680a1e729e1c2d81a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45111 } }
I20260810 03:24:38.791639 10652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:38.791762 10652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ff71860ab54680a1e729e1c2d81a5f, State: Initialized, Role: FOLLOWER
I20260810 03:24:38.792452 10652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ff71860ab54680a1e729e1c2d81a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45111 } }
I20260810 03:24:38.792804 10652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:38.792979 10652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:38.793179 10652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:38.795625 10652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ff71860ab54680a1e729e1c2d81a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45111 } }
I20260810 03:24:38.796249 10652 leader_election.cc:302] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37ff71860ab54680a1e729e1c2d81a5f; no voters:
I20260810 03:24:38.796960 10652 leader_election.cc:288] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:38.797132 10662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:38.799675 10662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [term 1 LEADER]: Becoming Leader. State: Replica: 37ff71860ab54680a1e729e1c2d81a5f, State: Running, Role: LEADER
I20260810 03:24:38.800065 10652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:38.800375 10662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ff71860ab54680a1e729e1c2d81a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45111 } }
I20260810 03:24:38.805207 10663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37ff71860ab54680a1e729e1c2d81a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ff71860ab54680a1e729e1c2d81a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45111 } } }
I20260810 03:24:38.806115 10663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:38.805155 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37ff71860ab54680a1e729e1c2d81a5f. Latest consensus state: current_term: 1 leader_uuid: "37ff71860ab54680a1e729e1c2d81a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ff71860ab54680a1e729e1c2d81a5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45111 } } }
I20260810 03:24:38.806840 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:38.808956 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:38.797152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53105 (local address 127.0.0.1:45111)
0810 03:24:38.797325 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:38.797330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:38.799057 (+ 1727us) server_negotiation.cc:407] Connection header received
0810 03:24:38.799309 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:38.799317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:38.799426 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:38.799547 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:38.800336 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:38.801286 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:38.803271 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:38.804149 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:38.804308 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:38.806282 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:38.806291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:38.806296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:38.806324 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:38.806378 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:38.806406 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:38.806549 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:38.808542 (+ 1993us) server_negotiation.cc:299] Negotiation successful
0810 03:24:38.808616 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:24:38.818389 10672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:38.828223 10672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:38.859431 10672 catalog_manager.cc:1347] Generated new cluster ID: 6ef39e34552b4c938b608e41c45b911b
I20260810 03:24:38.859606 10672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:38.884181 10672 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:38.886310 10672 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:38.902655 10672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37ff71860ab54680a1e729e1c2d81a5f: Generated new TSK 0
I20260810 03:24:38.904124 10672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:38.917367 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10577
W20260810 03:24:38.919203 10647 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43347
W20260810 03:24:38.920679 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10577: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:24:38.920907 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 10577
W20260810 03:24:38.944205 10561 connection.cc:535] client connection to 127.0.0.1:45111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45111): Connection reset by peer (error 104)
2026-08-10T03:24:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:39 dist-test-slave-btw8 krb5kdc[10688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:39 dist-test-slave-btw8 krb5kdc[10688](info): set up 2 sockets
Aug 10 03:24:39 dist-test-slave-btw8 krb5kdc[10688](info): commencing operation
krb5kdc: starting...
W20260810 03:24:40.196574 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.000s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:40 dist-test-slave-btw8 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:40.588950 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:40.979591 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:40.979998 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:40.980198 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:40.988914 10702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:40.989085 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:40.989183 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:40.989271 10702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:40.989355 10702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:41.001205 10702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:41.002979 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:41.004686 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:41.019169 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:41.019094 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:41.019063 10711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:41.020308 10702 server_base.cc:1034] running on GCE node
I20260810 03:24:41.021306 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:41.023049 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:41.024358 10702 hybrid_clock.cc:648] HybridClock initialized: now 1786332281024291 us; error 58 us; skew 500 ppm
I20260810 03:24:41.024829 10702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:41 dist-test-slave-btw8 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:41.032397 10702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:41.034332 10702 webserver.cc:460] Webserver started at http://127.0.0.1:44945/ using document root <none> and password file <none>
I20260810 03:24:41.035197 10702 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:41.035454 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:41.035836 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:41.039480 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "b881c23ddd6a48f9b72df26ec514b0c6"
format_stamp: "Formatted at 2026-08-10 03:24:41 on dist-test-slave-btw8"
server_key: "b0741bdedaa953ac7f121912412da129"
server_key_iv: "d832e200dac14171bc2429505d9eb478"
server_key_version: "encryptionkey@0"
I20260810 03:24:41.040426 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "b881c23ddd6a48f9b72df26ec514b0c6"
format_stamp: "Formatted at 2026-08-10 03:24:41 on dist-test-slave-btw8"
server_key: "b0741bdedaa953ac7f121912412da129"
server_key_iv: "d832e200dac14171bc2429505d9eb478"
server_key_version: "encryptionkey@0"
I20260810 03:24:41.046703 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 03:24:41.051039 10721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:41.052778 10702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:24:41.053018 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "b881c23ddd6a48f9b72df26ec514b0c6"
format_stamp: "Formatted at 2026-08-10 03:24:41 on dist-test-slave-btw8"
server_key: "b0741bdedaa953ac7f121912412da129"
server_key_iv: "d832e200dac14171bc2429505d9eb478"
server_key_version: "encryptionkey@0"
I20260810 03:24:41.053265 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:41.060279 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:41.065407 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:41.065814 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:41.087662 10702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34803
I20260810 03:24:41.087761 10780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34803 every 8 connection(s)
I20260810 03:24:41.089330 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:41.093926 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:41.099455 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 10702
I20260810 03:24:41.099676 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:41.100303 20443 external_mini_cluster.cc:1428] Setting key 9a5e31f4f0837986553833386b078b03
I20260810 03:24:41.108359 10781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6: Bootstrap starting.
I20260810 03:24:41.110199 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:41.102647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43273 (local address 127.0.0.1:34803)
0810 03:24:41.103666 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:41.103679 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:41.103731 (+ 52us) server_negotiation.cc:407] Connection header received
0810 03:24:41.106201 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:41.106232 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:41.106650 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:41.107126 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:41.108404 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:41.108411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:41.108424 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:41.108439 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:41.108497 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:41.108550 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:41.108817 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:41.108961 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 03:24:41.109183 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":432,"threads_started":1}
I20260810 03:24:41.112874 10781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:41.114265 10781 log.cc:824] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:41.115127 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:41.118238 10781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6: No bootstrap required, opened a new log
I20260810 03:24:41.124802 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:41.121363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33183 (local address 127.0.0.1:34803)
0810 03:24:41.121632 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:41.121644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:41.122104 (+ 460us) server_negotiation.cc:407] Connection header received
0810 03:24:41.122311 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:41.122317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:41.122388 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:41.122573 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:41.123765 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:41.123769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:41.123773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:41.123791 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:41.123840 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:41.123857 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:41.123968 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:41.124193 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 03:24:41.124385 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:24:41.128787 10781 raft_consensus.cc:348] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b881c23ddd6a48f9b72df26ec514b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } }
I20260810 03:24:41.129218 10781 raft_consensus.cc:374] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:41.129292 10781 raft_consensus.cc:729] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b881c23ddd6a48f9b72df26ec514b0c6, State: Initialized, Role: FOLLOWER
I20260810 03:24:41.130155 10781 consensus_queue.cc:260] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b881c23ddd6a48f9b72df26ec514b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } }
I20260810 03:24:41.130527 10781 raft_consensus.cc:388] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:41.130685 10781 raft_consensus.cc:482] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:41.130842 10781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:41.133523 10781 raft_consensus.cc:504] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b881c23ddd6a48f9b72df26ec514b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } }
I20260810 03:24:41.134130 10781 leader_election.cc:302] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b881c23ddd6a48f9b72df26ec514b0c6; no voters:
I20260810 03:24:41.134824 10781 leader_election.cc:288] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:41.134976 10792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:41.135973 10792 raft_consensus.cc:686] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [term 1 LEADER]: Becoming Leader. State: Replica: b881c23ddd6a48f9b72df26ec514b0c6, State: Running, Role: LEADER
I20260810 03:24:41.136655 10792 consensus_queue.cc:237] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b881c23ddd6a48f9b72df26ec514b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } }
I20260810 03:24:41.137367 10781 sys_catalog.cc:564] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:41.139830 10793 sys_catalog.cc:455] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b881c23ddd6a48f9b72df26ec514b0c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b881c23ddd6a48f9b72df26ec514b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } } }
I20260810 03:24:41.139959 10794 sys_catalog.cc:455] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b881c23ddd6a48f9b72df26ec514b0c6. Latest consensus state: current_term: 1 leader_uuid: "b881c23ddd6a48f9b72df26ec514b0c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b881c23ddd6a48f9b72df26ec514b0c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34803 } } }
I20260810 03:24:41.140419 10793 sys_catalog.cc:458] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:41.140540 10794 sys_catalog.cc:458] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:41.142282 10797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:41.148311 10797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:41.158324 10797 catalog_manager.cc:1347] Generated new cluster ID: 2fcea3745a4948fe95d3e6bbb65426ba
I20260810 03:24:41.158468 10797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:41.179121 10797 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:41.179800 10797 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:41.188732 10797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b881c23ddd6a48f9b72df26ec514b0c6: Generated new TSK 0
I20260810 03:24:41.189718 10797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:41.196051 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10702
W20260810 03:24:41.197558 10776 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43273
W20260810 03:24:41.198402 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10702: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:24:41.198587 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 10702
2026-08-10T03:24:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:41 dist-test-slave-btw8 krb5kdc[10815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:41 dist-test-slave-btw8 krb5kdc[10815](info): set up 2 sockets
Aug 10 03:24:41 dist-test-slave-btw8 krb5kdc[10815](info): commencing operation
krb5kdc: starting...
W20260810 03:24:42.437994 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:42 dist-test-slave-btw8 krb5kdc[10815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:42.821521 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:43.195791 10830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:43.196134 10830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:43.196297 10830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:43.203979 10830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:43.204126 10830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:43.204202 10830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:43.204305 10830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:43.204348 10830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:43.214960 10830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:43.216583 10830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:43.217970 10830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:43.232971 10835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:43.232892 10836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:43.232949 10838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:43.234287 10830 server_base.cc:1034] running on GCE node
I20260810 03:24:43.235442 10830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:43.237329 10830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:43.238679 10830 hybrid_clock.cc:648] HybridClock initialized: now 1786332283238635 us; error 121 us; skew 500 ppm
I20260810 03:24:43.239179 10830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:43 dist-test-slave-btw8 krb5kdc[10815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:43.242767 10830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:43.244549 10830 webserver.cc:460] Webserver started at http://127.0.0.1:46493/ using document root <none> and password file <none>
I20260810 03:24:43.245326 10830 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:43.245445 10830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:43.245733 10830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:43.249253 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "14a9b63969ca479084270460d2cc2cd1"
format_stamp: "Formatted at 2026-08-10 03:24:43 on dist-test-slave-btw8"
server_key: "bea12b12952135fb4274eb8f7da2cab6"
server_key_iv: "38d0277da398ea28b30d1161559038cf"
server_key_version: "encryptionkey@0"
I20260810 03:24:43.250128 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "14a9b63969ca479084270460d2cc2cd1"
format_stamp: "Formatted at 2026-08-10 03:24:43 on dist-test-slave-btw8"
server_key: "bea12b12952135fb4274eb8f7da2cab6"
server_key_iv: "38d0277da398ea28b30d1161559038cf"
server_key_version: "encryptionkey@0"
I20260810 03:24:43.256440 10830 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 03:24:43.260963 10845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:43.262784 10830 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:24:43.263091 10830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "14a9b63969ca479084270460d2cc2cd1"
format_stamp: "Formatted at 2026-08-10 03:24:43 on dist-test-slave-btw8"
server_key: "bea12b12952135fb4274eb8f7da2cab6"
server_key_iv: "38d0277da398ea28b30d1161559038cf"
server_key_version: "encryptionkey@0"
I20260810 03:24:43.263335 10830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:43.284770 10830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:43.289919 10830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:43.290271 10830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:43.311676 10830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41673
I20260810 03:24:43.311705 10905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41673 every 8 connection(s)
I20260810 03:24:43.313241 10830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:43.317538 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:43.320878 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 10830
I20260810 03:24:43.321102 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:43.321676 20443 external_mini_cluster.cc:1428] Setting key 948b0138bf0b1fd1685ec1a55788e09c
I20260810 03:24:43.331634 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1: Bootstrap starting.
I20260810 03:24:43.336002 10906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:43.337540 10906 log.cc:824] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:24:43 dist-test-slave-btw8 krb5kdc[10815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:24:43.341697 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1: No bootstrap required, opened a new log
I20260810 03:24:43.350165 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:43.323950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54453 (local address 127.0.0.1:41673)
0810 03:24:43.324932 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:43.324951 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:43.324999 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:24:43.326882 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:43.326908 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:43.327480 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:43.327880 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:43.329856 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:43.331304 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:43.333321 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:43.334181 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:43.334348 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:43.340670 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:43.340681 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:43.340699 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:43.340750 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:43.344823 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:43.346000 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:43.346009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:43.346017 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:43.346267 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:43.346797 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:43.346804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:43.346807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:43.347604 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:43.347900 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:43.348407 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 03:24:43.348965 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":480,"threads_started":1}
I20260810 03:24:43.351486 10906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9b63969ca479084270460d2cc2cd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41673 } }
I20260810 03:24:43.351934 10906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:43.352049 10906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a9b63969ca479084270460d2cc2cd1, State: Initialized, Role: FOLLOWER
I20260810 03:24:43.352703 10906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9b63969ca479084270460d2cc2cd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41673 } }
I20260810 03:24:43.353014 10906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:43.353168 10906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:43.353374 10906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:43.354960 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:43.356176 10906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9b63969ca479084270460d2cc2cd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41673 } }
I20260810 03:24:43.356753 10906 leader_election.cc:302] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14a9b63969ca479084270460d2cc2cd1; no voters:
I20260810 03:24:43.357425 10906 leader_election.cc:288] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:43.357646 10911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:43.359239 10911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [term 1 LEADER]: Becoming Leader. State: Replica: 14a9b63969ca479084270460d2cc2cd1, State: Running, Role: LEADER
I20260810 03:24:43.360162 10906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:43.360110 10911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9b63969ca479084270460d2cc2cd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41673 } }
I20260810 03:24:43.363478 10914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14a9b63969ca479084270460d2cc2cd1. Latest consensus state: current_term: 1 leader_uuid: "14a9b63969ca479084270460d2cc2cd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9b63969ca479084270460d2cc2cd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41673 } } }
I20260810 03:24:43.363917 10914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:43.364547 10912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14a9b63969ca479084270460d2cc2cd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a9b63969ca479084270460d2cc2cd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41673 } } }
I20260810 03:24:43.364941 10912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:43.366559 10925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:43.373993 10925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:43.387125 10925 catalog_manager.cc:1347] Generated new cluster ID: 0081241fc8a04d9c8c9d07d0bd0b2fe8
I20260810 03:24:43.387419 10925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:43.389358 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:43.362374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37119 (local address 127.0.0.1:41673)
0810 03:24:43.362632 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:43.362641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:43.366919 (+ 4278us) server_negotiation.cc:407] Connection header received
0810 03:24:43.367269 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:43.367275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:43.367399 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:43.367484 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:43.370310 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:43.371578 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:43.373910 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:43.374610 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:43.374736 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:43.379071 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:43.379086 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:43.379107 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:43.379183 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:43.385270 (+ 6087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:43.386518 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:43.386524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:43.386528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:43.386750 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:43.387332 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:43.387338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:43.387344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:43.387844 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:43.388167 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:43.388665 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 03:24:43.388938 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:24:43.410514 10925 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:43.412779 10925 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:43.425035 10925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14a9b63969ca479084270460d2cc2cd1: Generated new TSK 0
I20260810 03:24:43.425971 10925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:43.438445 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10830
I20260810 03:24:43.439941 10901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:43.537227 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 10830
2026-08-10T03:24:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:43 dist-test-slave-btw8 krb5kdc[10945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:43 dist-test-slave-btw8 krb5kdc[10945](info): set up 2 sockets
Aug 10 03:24:43 dist-test-slave-btw8 krb5kdc[10945](info): commencing operation
krb5kdc: starting...
W20260810 03:24:44.778620 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:45 dist-test-slave-btw8 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:45.155849 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:45.530200 10960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:45.530555 10960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:45.530656 10960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:45.538452 10960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:45.538605 10960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:45.538656 10960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:45.538712 10960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:45.538759 10960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:45.549711 10960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:45.551101 10960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:45.552517 10960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:45.565680 10967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:45.565773 10969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:45.565807 10965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:45.566578 10960 server_base.cc:1034] running on GCE node
I20260810 03:24:45.567520 10960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:45.570257 10960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:45.571496 10960 hybrid_clock.cc:648] HybridClock initialized: now 1786332285571438 us; error 44 us; skew 500 ppm
I20260810 03:24:45.571926 10960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:45 dist-test-slave-btw8 krb5kdc[10945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:45.575459 10960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:45.577157 10960 webserver.cc:460] Webserver started at http://127.0.0.1:41299/ using document root <none> and password file <none>
I20260810 03:24:45.578039 10960 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:45.578166 10960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:45.578454 10960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:45.581748 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "c3b3dc44abef426da4a9970b59d13712"
format_stamp: "Formatted at 2026-08-10 03:24:45 on dist-test-slave-btw8"
server_key: "fd9e548d7e62f66fe6b5341825a2105a"
server_key_iv: "e1365783b7a5cfd46bd90861a26f39ab"
server_key_version: "encryptionkey@0"
I20260810 03:24:45.582551 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "c3b3dc44abef426da4a9970b59d13712"
format_stamp: "Formatted at 2026-08-10 03:24:45 on dist-test-slave-btw8"
server_key: "fd9e548d7e62f66fe6b5341825a2105a"
server_key_iv: "e1365783b7a5cfd46bd90861a26f39ab"
server_key_version: "encryptionkey@0"
I20260810 03:24:45.588449 10960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 03:24:45.592715 10976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:45.594228 10960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 03:24:45.594441 10960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "c3b3dc44abef426da4a9970b59d13712"
format_stamp: "Formatted at 2026-08-10 03:24:45 on dist-test-slave-btw8"
server_key: "fd9e548d7e62f66fe6b5341825a2105a"
server_key_iv: "e1365783b7a5cfd46bd90861a26f39ab"
server_key_version: "encryptionkey@0"
I20260810 03:24:45.594640 10960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:45.610335 10960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:45.615091 10960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:45.615483 10960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:45.636694 10960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37325
I20260810 03:24:45.636749 11036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37325 every 8 connection(s)
I20260810 03:24:45.638227 10960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:45.642477 11037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:45.645576 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 10960
I20260810 03:24:45.645785 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:45.646334 20443 external_mini_cluster.cc:1428] Setting key d7b47ea75448dc45cc9f1e320f883a70
I20260810 03:24:45.656373 11037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712: Bootstrap starting.
I20260810 03:24:45.660563 11037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:45.661873 11037 log.cc:824] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:45.662595 11041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:45.648588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40961 (local address 127.0.0.1:37325)
0810 03:24:45.649543 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:45.649556 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:45.649602 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:24:45.651480 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:45.651506 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:45.651977 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:45.652362 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:45.653132 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:45.654974 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:45.657190 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:45.657949 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:45.658125 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:45.659654 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:45.659663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:45.659678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:45.660290 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:45.660380 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:45.660407 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:45.660525 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:45.661022 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 03:24:45.661278 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":465,"threads_started":1}
I20260810 03:24:45.665498 11037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712: No bootstrap required, opened a new log
I20260810 03:24:45.667058 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:45.676473 11037 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b3dc44abef426da4a9970b59d13712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260810 03:24:45.677223 11037 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:45.677383 11037 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b3dc44abef426da4a9970b59d13712, State: Initialized, Role: FOLLOWER
I20260810 03:24:45.678730 11037 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b3dc44abef426da4a9970b59d13712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260810 03:24:45.679293 11037 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:45.679536 11037 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:45.679801 11037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:45.681488 11041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:45.672255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48487 (local address 127.0.0.1:37325)
0810 03:24:45.672511 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:45.672518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:45.672863 (+ 345us) server_negotiation.cc:407] Connection header received
0810 03:24:45.673093 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:45.673097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:45.673168 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:45.673281 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:45.673937 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:45.674746 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:45.677061 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:45.678622 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:45.678835 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:45.680535 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:45.680541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:45.680544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:45.680567 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:45.680661 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:45.680685 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:45.680792 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:45.680997 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 03:24:45.681108 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 03:24:45.683717 11037 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b3dc44abef426da4a9970b59d13712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260810 03:24:45.684367 11037 leader_election.cc:302] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3b3dc44abef426da4a9970b59d13712; no voters:
I20260810 03:24:45.685079 11037 leader_election.cc:288] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:45.685215 11049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:45.686865 11049 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [term 1 LEADER]: Becoming Leader. State: Replica: c3b3dc44abef426da4a9970b59d13712, State: Running, Role: LEADER
I20260810 03:24:45.687563 11049 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b3dc44abef426da4a9970b59d13712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260810 03:24:45.687893 11037 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:45.691125 11050 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3b3dc44abef426da4a9970b59d13712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b3dc44abef426da4a9970b59d13712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } } }
I20260810 03:24:45.694139 11050 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:45.693272 11052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b3dc44abef426da4a9970b59d13712. Latest consensus state: current_term: 1 leader_uuid: "c3b3dc44abef426da4a9970b59d13712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b3dc44abef426da4a9970b59d13712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } } }
I20260810 03:24:45.694674 11052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:45.696827 11059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:45.704203 11059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:45.716413 11059 catalog_manager.cc:1347] Generated new cluster ID: 7f8e69423f49426e8dd611401d37b086
I20260810 03:24:45.716559 11059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:45.732908 11059 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:45.734349 11059 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:45.745527 11059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3b3dc44abef426da4a9970b59d13712: Generated new TSK 0
I20260810 03:24:45.746541 11059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:45.755317 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10960
W20260810 03:24:45.756706 11032 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40961
W20260810 03:24:45.757587 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 10960: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:24:45.757719 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 10960
2026-08-10T03:24:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:45 dist-test-slave-btw8 krb5kdc[11072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:45 dist-test-slave-btw8 krb5kdc[11072](info): set up 2 sockets
Aug 10 03:24:45 dist-test-slave-btw8 krb5kdc[11072](info): commencing operation
krb5kdc: starting...
W20260810 03:24:47.001338 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:47 dist-test-slave-btw8 krb5kdc[11072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:47.397261 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:47.781601 11086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:47.781967 11086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:47.782078 11086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:47.790345 11086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:47.790513 11086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:47.790571 11086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:47.790629 11086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:47.790688 11086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:47.802156 11086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:47.803546 11086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:47.804823 11086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:47.817931 11091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:47.817888 11092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:47.817831 11094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:47.818846 11086 server_base.cc:1034] running on GCE node
I20260810 03:24:47.819959 11086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:47.821971 11086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:47.823246 11086 hybrid_clock.cc:648] HybridClock initialized: now 1786332287823195 us; error 62 us; skew 500 ppm
I20260810 03:24:47.823668 11086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:47 dist-test-slave-btw8 krb5kdc[11072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:47.833822 11086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:47.835616 11086 webserver.cc:460] Webserver started at http://127.0.0.1:34277/ using document root <none> and password file <none>
I20260810 03:24:47.836318 11086 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:47.836426 11086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:47.836699 11086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:47.839874 11086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "6cea4ebed7e442d88dc61b0beb934bdb"
format_stamp: "Formatted at 2026-08-10 03:24:47 on dist-test-slave-btw8"
server_key: "a84a965db9262cdec05d4054283fd277"
server_key_iv: "c8235c8dd2a303b935ae1b05a1d8d35d"
server_key_version: "encryptionkey@0"
I20260810 03:24:47.840634 11086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "6cea4ebed7e442d88dc61b0beb934bdb"
format_stamp: "Formatted at 2026-08-10 03:24:47 on dist-test-slave-btw8"
server_key: "a84a965db9262cdec05d4054283fd277"
server_key_iv: "c8235c8dd2a303b935ae1b05a1d8d35d"
server_key_version: "encryptionkey@0"
I20260810 03:24:47.846191 11086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260810 03:24:47.850273 11102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:47.852049 11086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 03:24:47.852228 11086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "6cea4ebed7e442d88dc61b0beb934bdb"
format_stamp: "Formatted at 2026-08-10 03:24:47 on dist-test-slave-btw8"
server_key: "a84a965db9262cdec05d4054283fd277"
server_key_iv: "c8235c8dd2a303b935ae1b05a1d8d35d"
server_key_version: "encryptionkey@0"
I20260810 03:24:47.852437 11086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:47.858873 11086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:47.863708 11086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:47.864032 11086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:47.885752 11086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43559
I20260810 03:24:47.885926 11158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43559 every 8 connection(s)
I20260810 03:24:47.887426 11086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:47.892041 11159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:47.896665 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 11086
I20260810 03:24:47.896869 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:47.897372 20443 external_mini_cluster.cc:1428] Setting key 8260bc77930c06f4ea776a7e0215f85d
I20260810 03:24:47.906339 11159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb: Bootstrap starting.
I20260810 03:24:47.908795 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:47.899406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60063 (local address 127.0.0.1:43559)
0810 03:24:47.900570 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:47.900587 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:47.900647 (+ 60us) server_negotiation.cc:407] Connection header received
0810 03:24:47.903835 (+ 3188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:47.903872 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:47.904401 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:47.904863 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:47.906267 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:47.906276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:47.906333 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:47.906357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:47.906447 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:47.906480 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:47.906795 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:47.906949 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 03:24:47.907171 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":500,"threads_started":1}
I20260810 03:24:47.910871 11159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:47.912492 11159 log.cc:824] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:47.913894 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:47.916402 11159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb: No bootstrap required, opened a new log
I20260810 03:24:47.923169 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:47.920141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56171 (local address 127.0.0.1:43559)
0810 03:24:47.920408 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:47.920414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:47.920807 (+ 393us) server_negotiation.cc:407] Connection header received
0810 03:24:47.920986 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:47.920992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:47.921059 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:47.921200 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:47.922371 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:47.922377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:47.922382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:47.922403 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:47.922460 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:47.922489 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:47.922628 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:47.922774 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 03:24:47.922845 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260810 03:24:47.927119 11159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea4ebed7e442d88dc61b0beb934bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260810 03:24:47.927699 11159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:47.927817 11159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cea4ebed7e442d88dc61b0beb934bdb, State: Initialized, Role: FOLLOWER
I20260810 03:24:47.928480 11159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea4ebed7e442d88dc61b0beb934bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260810 03:24:47.928843 11159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:47.929016 11159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:47.929222 11159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:47.932051 11159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea4ebed7e442d88dc61b0beb934bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260810 03:24:47.932660 11159 leader_election.cc:302] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cea4ebed7e442d88dc61b0beb934bdb; no voters:
I20260810 03:24:47.933403 11159 leader_election.cc:288] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:47.933576 11169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:47.934304 11169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [term 1 LEADER]: Becoming Leader. State: Replica: 6cea4ebed7e442d88dc61b0beb934bdb, State: Running, Role: LEADER
I20260810 03:24:47.934989 11169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea4ebed7e442d88dc61b0beb934bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } }
I20260810 03:24:47.936007 11159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:47.938519 11170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cea4ebed7e442d88dc61b0beb934bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea4ebed7e442d88dc61b0beb934bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } } }
I20260810 03:24:47.938987 11170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:47.938441 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cea4ebed7e442d88dc61b0beb934bdb. Latest consensus state: current_term: 1 leader_uuid: "6cea4ebed7e442d88dc61b0beb934bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea4ebed7e442d88dc61b0beb934bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43559 } } }
I20260810 03:24:47.939591 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:47.943266 11178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:47.948060 11178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:47.958020 11178 catalog_manager.cc:1347] Generated new cluster ID: 9d724653e585403880821518e579e780
I20260810 03:24:47.958187 11178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:47.996517 11178 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:47.997148 11178 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:48.013693 11178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cea4ebed7e442d88dc61b0beb934bdb: Generated new TSK 0
I20260810 03:24:48.014950 11178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:48.032595 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11086
W20260810 03:24:48.033882 11153 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60063
W20260810 03:24:48.034811 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11086: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:24:48.034977 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 11086
2026-08-10T03:24:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:48 dist-test-slave-btw8 krb5kdc[11191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:48 dist-test-slave-btw8 krb5kdc[11191](info): set up 2 sockets
Aug 10 03:24:48 dist-test-slave-btw8 krb5kdc[11191](info): commencing operation
krb5kdc: starting...
W20260810 03:24:49.292604 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.001s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:49 dist-test-slave-btw8 krb5kdc[11191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:49.688323 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:50.079341 11207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:50.079790 11207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:50.079981 11207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:50.088963 11207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:50.089114 11207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:50.089218 11207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:50.089313 11207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:50.089407 11207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:50.100306 11207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:50.101727 11207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:50.103266 11207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:50.117129 11216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:50.117262 11214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:50.117244 11213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:50.118022 11207 server_base.cc:1034] running on GCE node
I20260810 03:24:50.118912 11207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:50.120617 11207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:50.121853 11207 hybrid_clock.cc:648] HybridClock initialized: now 1786332290121812 us; error 35 us; skew 500 ppm
I20260810 03:24:50.122303 11207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:50 dist-test-slave-btw8 krb5kdc[11191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:50.126050 11207 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:50.127882 11207 webserver.cc:460] Webserver started at http://127.0.0.1:36503/ using document root <none> and password file <none>
I20260810 03:24:50.128657 11207 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:50.128801 11207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:50.129160 11207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:50.132531 11207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "070133c275ac4ab892cadaeb4c55e8e9"
format_stamp: "Formatted at 2026-08-10 03:24:50 on dist-test-slave-btw8"
server_key: "db44b5afca3e67d50a700bb5024612e1"
server_key_iv: "90e3eb311333b906a816149c93dff2b4"
server_key_version: "encryptionkey@0"
I20260810 03:24:50.133422 11207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "070133c275ac4ab892cadaeb4c55e8e9"
format_stamp: "Formatted at 2026-08-10 03:24:50 on dist-test-slave-btw8"
server_key: "db44b5afca3e67d50a700bb5024612e1"
server_key_iv: "90e3eb311333b906a816149c93dff2b4"
server_key_version: "encryptionkey@0"
I20260810 03:24:50.139526 11207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 03:24:50.143888 11224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:50.145460 11207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:24:50.145764 11207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "070133c275ac4ab892cadaeb4c55e8e9"
format_stamp: "Formatted at 2026-08-10 03:24:50 on dist-test-slave-btw8"
server_key: "db44b5afca3e67d50a700bb5024612e1"
server_key_iv: "90e3eb311333b906a816149c93dff2b4"
server_key_version: "encryptionkey@0"
I20260810 03:24:50.146018 11207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:50.177104 11207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:50.182717 11207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:50.183097 11207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:50.205170 11207 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44431
I20260810 03:24:50.205202 11279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44431 every 8 connection(s)
I20260810 03:24:50.206686 11207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:50.208479 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 11207
I20260810 03:24:50.208745 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:50.209272 20443 external_mini_cluster.cc:1428] Setting key f16e9f85e0144dff205a219f286c38cb
I20260810 03:24:50.211216 11280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:24:50 dist-test-slave-btw8 krb5kdc[11191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:24:50.226068 11280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9: Bootstrap starting.
I20260810 03:24:50.229681 11280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:50.231004 11280 log.cc:824] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:50.234676 11280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9: No bootstrap required, opened a new log
I20260810 03:24:50.235595 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:50.211058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:49877 (local address 127.0.0.1:44431)
0810 03:24:50.211943 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:50.211955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:50.211995 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:24:50.213658 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:50.213679 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:50.214074 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:50.214418 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:50.216362 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:50.217808 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:50.219652 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:50.220437 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:50.220591 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:50.226121 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:50.226132 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:50.226187 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:50.226256 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:50.230353 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:50.231536 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:50.231543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:50.231550 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:50.231900 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:50.232449 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:50.232455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:50.232458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:50.233077 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:50.233333 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:50.233925 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 03:24:50.234280 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":442,"threads_started":1}
I20260810 03:24:50.240324 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:50.245905 11280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070133c275ac4ab892cadaeb4c55e8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } }
I20260810 03:24:50.246472 11280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:50.246665 11280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070133c275ac4ab892cadaeb4c55e8e9, State: Initialized, Role: FOLLOWER
I20260810 03:24:50.247538 11280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070133c275ac4ab892cadaeb4c55e8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } }
I20260810 03:24:50.247869 11280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:50.248024 11280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:50.248253 11280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:50.251547 11280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070133c275ac4ab892cadaeb4c55e8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } }
I20260810 03:24:50.252223 11280 leader_election.cc:302] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070133c275ac4ab892cadaeb4c55e8e9; no voters:
I20260810 03:24:50.253046 11280 leader_election.cc:288] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:50.253199 11292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:50.255374 11292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [term 1 LEADER]: Becoming Leader. State: Replica: 070133c275ac4ab892cadaeb4c55e8e9, State: Running, Role: LEADER
I20260810 03:24:50.257867 11292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070133c275ac4ab892cadaeb4c55e8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } }
I20260810 03:24:50.260231 11280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:50.262025 11294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070133c275ac4ab892cadaeb4c55e8e9. Latest consensus state: current_term: 1 leader_uuid: "070133c275ac4ab892cadaeb4c55e8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070133c275ac4ab892cadaeb4c55e8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } } }
I20260810 03:24:50.262466 11294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:50.265520 11295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070133c275ac4ab892cadaeb4c55e8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070133c275ac4ab892cadaeb4c55e8e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44431 } } }
I20260810 03:24:50.266575 11295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:50.269207 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:50.246408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:34081 (local address 127.0.0.1:44431)
0810 03:24:50.246669 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:50.246676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:50.247076 (+ 400us) server_negotiation.cc:407] Connection header received
0810 03:24:50.247274 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:50.247282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:50.247416 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:50.247523 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:50.249724 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:50.250878 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:50.252775 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:50.253597 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:50.253832 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:50.257739 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:50.257750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:50.257758 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:50.257813 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:50.262481 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:50.264045 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:50.264053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:50.264059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:50.264331 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:50.265163 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:50.265172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:50.265178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:50.265654 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:50.268073 (+ 2419us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:50.268349 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 03:24:50.268632 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 03:24:50.273794 11298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:50.281252 11298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:50.293033 11298 catalog_manager.cc:1347] Generated new cluster ID: 8d1ac8f693234363a634cbf1eb9cdb76
I20260810 03:24:50.293191 11298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:50.308178 11298 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:50.309737 11298 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:50.322186 11298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070133c275ac4ab892cadaeb4c55e8e9: Generated new TSK 0
I20260810 03:24:50.323192 11298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:50.331875 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11207
I20260810 03:24:50.333923 11275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:50.433275 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 11207
W20260810 03:24:50.455960 11189 connection.cc:535] client connection to 127.0.0.1:44431 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44431): Connection reset by peer (error 104)
2026-08-10T03:24:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:50 dist-test-slave-btw8 krb5kdc[11316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:50 dist-test-slave-btw8 krb5kdc[11316](info): set up 2 sockets
Aug 10 03:24:50 dist-test-slave-btw8 krb5kdc[11316](info): commencing operation
krb5kdc: starting...
W20260810 03:24:51.691071 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.002s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:51 dist-test-slave-btw8 krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:52.092376 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:52.463723 11331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:52.464037 11331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:52.464143 11331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:52.471714 11331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:52.471853 11331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:52.471901 11331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:52.471958 11331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:52.472004 11331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:52.482357 11331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:52.483753 11331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:52.485116 11331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:52.499106 11336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:52.499053 11337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:52.499017 11339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:52.500191 11331 server_base.cc:1034] running on GCE node
I20260810 03:24:52.501188 11331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:52.502882 11331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:52.504235 11331 hybrid_clock.cc:648] HybridClock initialized: now 1786332292504194 us; error 75 us; skew 500 ppm
I20260810 03:24:52.504702 11331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:52 dist-test-slave-btw8 krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:52.508780 11331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:52.511039 11331 webserver.cc:460] Webserver started at http://127.0.0.1:38863/ using document root <none> and password file <none>
I20260810 03:24:52.512310 11331 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:52.512531 11331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:52.512907 11331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:52.516584 11331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "7fcf749d22a847838a052e12c64a8a4c"
format_stamp: "Formatted at 2026-08-10 03:24:52 on dist-test-slave-btw8"
server_key: "0453e8046ef084d93d08648d5625c8d6"
server_key_iv: "78e46c0c546ff6deb17a14f743a62c7c"
server_key_version: "encryptionkey@0"
I20260810 03:24:52.517475 11331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "7fcf749d22a847838a052e12c64a8a4c"
format_stamp: "Formatted at 2026-08-10 03:24:52 on dist-test-slave-btw8"
server_key: "0453e8046ef084d93d08648d5625c8d6"
server_key_iv: "78e46c0c546ff6deb17a14f743a62c7c"
server_key_version: "encryptionkey@0"
I20260810 03:24:52.523825 11331 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 03:24:52.527953 11347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:52.529897 11331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:24:52.530139 11331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "7fcf749d22a847838a052e12c64a8a4c"
format_stamp: "Formatted at 2026-08-10 03:24:52 on dist-test-slave-btw8"
server_key: "0453e8046ef084d93d08648d5625c8d6"
server_key_iv: "78e46c0c546ff6deb17a14f743a62c7c"
server_key_version: "encryptionkey@0"
I20260810 03:24:52.530589 11331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:52.544456 11331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:52.549813 11331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:52.550248 11331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:52.572517 11331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34571
I20260810 03:24:52.572536 11403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34571 every 8 connection(s)
I20260810 03:24:52.574087 11331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:52.578337 11404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:52.581410 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 11331
I20260810 03:24:52.581658 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:52.582297 20443 external_mini_cluster.cc:1428] Setting key 2e79c22e44daaef317224ea77c0fe2fc
I20260810 03:24:52.593614 11404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c: Bootstrap starting.
I20260810 03:24:52.597484 11404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:52.598922 11404 log.cc:824] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:52.599726 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:52.585119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:33207 (local address 127.0.0.1:34571)
0810 03:24:52.586255 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:52.586271 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:52.586329 (+ 58us) server_negotiation.cc:407] Connection header received
0810 03:24:52.588661 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:52.588702 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:52.589252 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:52.589748 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:52.590411 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:52.592232 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:52.593907 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:52.594771 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:52.595026 (+ 255us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:52.596658 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:52.596679 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:52.596696 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:52.597358 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:52.597455 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:52.597494 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:52.597703 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:52.598039 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 03:24:52.598254 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":524,"threads_started":1}
I20260810 03:24:52.602694 11404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c: No bootstrap required, opened a new log
I20260810 03:24:52.604940 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:52.616037 11404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcf749d22a847838a052e12c64a8a4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34571 } }
I20260810 03:24:52.616741 11404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:52.616905 11404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fcf749d22a847838a052e12c64a8a4c, State: Initialized, Role: FOLLOWER
I20260810 03:24:52.617704 11404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcf749d22a847838a052e12c64a8a4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34571 } }
I20260810 03:24:52.618019 11404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:52.618217 11404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:52.618429 11404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:52.620033 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:52.611674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:37251 (local address 127.0.0.1:34571)
0810 03:24:52.611958 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:52.611968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:52.612430 (+ 462us) server_negotiation.cc:407] Connection header received
0810 03:24:52.612674 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:52.612680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:52.612759 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:52.612861 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:52.613658 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:52.614495 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:52.616651 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:52.617289 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:52.617465 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:52.619150 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:52.619155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:52.619159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:52.619184 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:52.619263 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:52.619281 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:52.619465 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:52.619664 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 03:24:52.619747 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:24:52.621685 11404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcf749d22a847838a052e12c64a8a4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34571 } }
I20260810 03:24:52.622323 11404 leader_election.cc:302] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fcf749d22a847838a052e12c64a8a4c; no voters:
I20260810 03:24:52.623050 11404 leader_election.cc:288] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:52.623250 11417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:52.625579 11404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:52.627619 11417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [term 1 LEADER]: Becoming Leader. State: Replica: 7fcf749d22a847838a052e12c64a8a4c, State: Running, Role: LEADER
I20260810 03:24:52.628270 11417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcf749d22a847838a052e12c64a8a4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34571 } }
I20260810 03:24:52.631460 11420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fcf749d22a847838a052e12c64a8a4c. Latest consensus state: current_term: 1 leader_uuid: "7fcf749d22a847838a052e12c64a8a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcf749d22a847838a052e12c64a8a4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34571 } } }
I20260810 03:24:52.631968 11420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:52.632615 11419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fcf749d22a847838a052e12c64a8a4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fcf749d22a847838a052e12c64a8a4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34571 } } }
I20260810 03:24:52.632979 11419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:52.634939 11428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:52.641294 11428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:52.651965 11428 catalog_manager.cc:1347] Generated new cluster ID: 1845b75ff5bf415bb66789f2a77abdad
I20260810 03:24:52.652127 11428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:52.664798 11428 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:52.666363 11428 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:52.676815 11428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fcf749d22a847838a052e12c64a8a4c: Generated new TSK 0
I20260810 03:24:52.677829 11428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:52.685607 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11331
W20260810 03:24:52.686950 11399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:33207
W20260810 03:24:52.687955 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11331: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:24:52.688166 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 11331
W20260810 03:24:52.708035 11314 connection.cc:535] client connection to 127.0.0.1:34571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34571): Connection reset by peer (error 104)
2026-08-10T03:24:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:52 dist-test-slave-btw8 krb5kdc[11445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:52 dist-test-slave-btw8 krb5kdc[11445](info): set up 2 sockets
Aug 10 03:24:52 dist-test-slave-btw8 krb5kdc[11445](info): commencing operation
krb5kdc: starting...
W20260810 03:24:53.971905 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.006s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:54 dist-test-slave-btw8 krb5kdc[11445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:54.368566 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:54.758337 11458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:54.758667 11458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:54.758827 11458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:54.766407 11458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:54.766546 11458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:54.766636 11458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:54.766738 11458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:54.766849 11458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:54.777261 11458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:54.778717 11458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:54.780238 11458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:54.795050 11466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:54.795037 11463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:54.795073 11464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:54.798062 11458 server_base.cc:1034] running on GCE node
I20260810 03:24:54.799058 11458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:54.800879 11458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:54.802157 11458 hybrid_clock.cc:648] HybridClock initialized: now 1786332294802119 us; error 43 us; skew 500 ppm
I20260810 03:24:54.802570 11458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:54 dist-test-slave-btw8 krb5kdc[11445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:54.816553 11458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:54.818691 11458 webserver.cc:460] Webserver started at http://127.0.0.1:35559/ using document root <none> and password file <none>
I20260810 03:24:54.819653 11458 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:54.819836 11458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:54.820197 11458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:54.823823 11458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "6ec042f6ae8d46399bf9a14936d6f4c4"
format_stamp: "Formatted at 2026-08-10 03:24:54 on dist-test-slave-btw8"
server_key: "0ceb55e86751d9c19061eec639da0906"
server_key_iv: "e84c32690b37ff51a443d1c1608f52f8"
server_key_version: "encryptionkey@0"
I20260810 03:24:54.824774 11458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "6ec042f6ae8d46399bf9a14936d6f4c4"
format_stamp: "Formatted at 2026-08-10 03:24:54 on dist-test-slave-btw8"
server_key: "0ceb55e86751d9c19061eec639da0906"
server_key_iv: "e84c32690b37ff51a443d1c1608f52f8"
server_key_version: "encryptionkey@0"
I20260810 03:24:54.831534 11458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 03:24:54.835992 11474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:54.837908 11458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:24:54.838197 11458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "6ec042f6ae8d46399bf9a14936d6f4c4"
format_stamp: "Formatted at 2026-08-10 03:24:54 on dist-test-slave-btw8"
server_key: "0ceb55e86751d9c19061eec639da0906"
server_key_iv: "e84c32690b37ff51a443d1c1608f52f8"
server_key_version: "encryptionkey@0"
I20260810 03:24:54.838480 11458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:54.846028 11458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:54.852165 11458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:54.852525 11458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:54.874200 11458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45387
I20260810 03:24:54.874251 11528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45387 every 8 connection(s)
I20260810 03:24:54.875778 11458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:54.878062 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 11458
I20260810 03:24:54.878261 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:54.878867 20443 external_mini_cluster.cc:1428] Setting key 26c17fc24d7bf3ebba4bc4ec13f0232c
I20260810 03:24:54.880883 11529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:54.888209 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:54.880988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:45217 (local address 127.0.0.1:45387)
0810 03:24:54.882034 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:54.882048 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:54.882102 (+ 54us) server_negotiation.cc:407] Connection header received
0810 03:24:54.884027 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:54.884055 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:54.884490 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:54.884910 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:54.886140 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:54.886147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:54.886162 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:54.886179 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:54.886270 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:54.886292 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:54.886506 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:54.886700 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 03:24:54.887006 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":446,"threads_started":1}
I20260810 03:24:54.892712 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:54.897019 11529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4: Bootstrap starting.
I20260810 03:24:54.902670 11529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:54.903193 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:54.899871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:59149 (local address 127.0.0.1:45387)
0810 03:24:54.900087 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:54.900098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:54.900634 (+ 536us) server_negotiation.cc:407] Connection header received
0810 03:24:54.900880 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:54.900884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:54.900942 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:54.901058 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:54.902331 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:54.902336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:54.902340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:54.902360 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:54.902410 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:54.902429 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:54.902553 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:54.902770 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 03:24:54.902900 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 03:24:54.904345 11529 log.cc:824] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:54.908054 11529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4: No bootstrap required, opened a new log
I20260810 03:24:54.918191 11529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec042f6ae8d46399bf9a14936d6f4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } }
I20260810 03:24:54.918838 11529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:54.918990 11529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec042f6ae8d46399bf9a14936d6f4c4, State: Initialized, Role: FOLLOWER
I20260810 03:24:54.919963 11529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec042f6ae8d46399bf9a14936d6f4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } }
I20260810 03:24:54.920327 11529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:54.920576 11529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:54.920817 11529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:54.923774 11529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec042f6ae8d46399bf9a14936d6f4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } }
I20260810 03:24:54.924475 11529 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ec042f6ae8d46399bf9a14936d6f4c4; no voters:
I20260810 03:24:54.925331 11529 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:54.925552 11540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:54.926365 11540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [term 1 LEADER]: Becoming Leader. State: Replica: 6ec042f6ae8d46399bf9a14936d6f4c4, State: Running, Role: LEADER
I20260810 03:24:54.927104 11540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec042f6ae8d46399bf9a14936d6f4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } }
I20260810 03:24:54.928275 11529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:54.930980 11541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ec042f6ae8d46399bf9a14936d6f4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec042f6ae8d46399bf9a14936d6f4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } } }
I20260810 03:24:54.932150 11541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:54.931579 11542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec042f6ae8d46399bf9a14936d6f4c4. Latest consensus state: current_term: 1 leader_uuid: "6ec042f6ae8d46399bf9a14936d6f4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec042f6ae8d46399bf9a14936d6f4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45387 } } }
I20260810 03:24:54.933015 11542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:54.933948 11549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:54.941176 11549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:54.953504 11549 catalog_manager.cc:1347] Generated new cluster ID: fd3e6f866cf3457f960a43aa9a6dc61c
I20260810 03:24:54.953670 11549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:54.968497 11549 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:54.969020 11549 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:54.978772 11549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ec042f6ae8d46399bf9a14936d6f4c4: Generated new TSK 0
I20260810 03:24:54.979929 11549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:54.993409 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11458
W20260810 03:24:54.994642 11523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:45217
W20260810 03:24:54.995429 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11458: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:24:54.995572 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 11458
2026-08-10T03:24:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:55 dist-test-slave-btw8 krb5kdc[11562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:55 dist-test-slave-btw8 krb5kdc[11562](info): set up 2 sockets
Aug 10 03:24:55 dist-test-slave-btw8 krb5kdc[11562](info): commencing operation
krb5kdc: starting...
W20260810 03:24:56.269953 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:56 dist-test-slave-btw8 krb5kdc[11562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:56.677248 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:57.066805 11576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:57.067147 11576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:57.067330 11576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:57.075209 11576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:57.075418 11576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:57.075520 11576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:57.075651 11576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:57.075739 11576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:57.086524 11576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:57.088102 11576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:57.089619 11576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:57.103837 11581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:57.103914 11582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:57.103873 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:57.105376 11576 server_base.cc:1034] running on GCE node
I20260810 03:24:57.106595 11576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:57.108547 11576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:57.109845 11576 hybrid_clock.cc:648] HybridClock initialized: now 1786332297109797 us; error 53 us; skew 500 ppm
I20260810 03:24:57.110322 11576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:57 dist-test-slave-btw8 krb5kdc[11562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:57.114427 11576 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:57.116631 11576 webserver.cc:460] Webserver started at http://127.0.0.1:33279/ using document root <none> and password file <none>
I20260810 03:24:57.117419 11576 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:57.117542 11576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:57.117879 11576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:57.121497 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "e090259a4f4047e78dc206ce9e169360"
format_stamp: "Formatted at 2026-08-10 03:24:57 on dist-test-slave-btw8"
server_key: "554e3f7d78e9a188572d1b836a1e7f62"
server_key_iv: "0ef63f1a0aa4aba81008b8c64700bde5"
server_key_version: "encryptionkey@0"
I20260810 03:24:57.122407 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "e090259a4f4047e78dc206ce9e169360"
format_stamp: "Formatted at 2026-08-10 03:24:57 on dist-test-slave-btw8"
server_key: "554e3f7d78e9a188572d1b836a1e7f62"
server_key_iv: "0ef63f1a0aa4aba81008b8c64700bde5"
server_key_version: "encryptionkey@0"
I20260810 03:24:57.128793 11576 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 03:24:57.134183 11593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:57.135963 11576 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:24:57.136214 11576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "e090259a4f4047e78dc206ce9e169360"
format_stamp: "Formatted at 2026-08-10 03:24:57 on dist-test-slave-btw8"
server_key: "554e3f7d78e9a188572d1b836a1e7f62"
server_key_iv: "0ef63f1a0aa4aba81008b8c64700bde5"
server_key_version: "encryptionkey@0"
I20260810 03:24:57.136469 11576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:57.154699 11576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:57.160893 11576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:57.161285 11576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:57.184176 11576 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38981
I20260810 03:24:57.184245 11647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38981 every 8 connection(s)
I20260810 03:24:57.185822 11576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:57.187624 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 11576
I20260810 03:24:57.187839 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:57.188434 20443 external_mini_cluster.cc:1428] Setting key 7f64155752c38ba27d0731a940345548
I20260810 03:24:57.191161 11648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:24:57 dist-test-slave-btw8 krb5kdc[11562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:24:57.206310 11648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360: Bootstrap starting.
I20260810 03:24:57.211228 11648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:57.212527 11648 log.cc:824] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:57.216204 11648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360: No bootstrap required, opened a new log
I20260810 03:24:57.216650 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:57.190813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:34497 (local address 127.0.0.1:38981)
0810 03:24:57.191921 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:57.191936 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:57.191984 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:24:57.193990 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:57.194016 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:57.194640 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:57.195150 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:57.196872 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:57.198372 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:57.199873 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:57.200677 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:57.200913 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:57.206299 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:57.206310 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:57.206328 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:57.206378 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:57.211081 (+ 4703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:57.212295 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:57.212304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:57.212311 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:57.212534 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:57.213148 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:57.213152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:57.213155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:57.213818 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:57.214408 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:57.215012 (+ 604us) server_negotiation.cc:299] Negotiation successful
0810 03:24:57.215457 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":567,"threads_started":1}
I20260810 03:24:57.221125 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:57.225358 11648 raft_consensus.cc:348] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090259a4f4047e78dc206ce9e169360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } }
I20260810 03:24:57.226038 11648 raft_consensus.cc:374] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:57.226217 11648 raft_consensus.cc:729] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e090259a4f4047e78dc206ce9e169360, State: Initialized, Role: FOLLOWER
I20260810 03:24:57.227012 11648 consensus_queue.cc:260] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090259a4f4047e78dc206ce9e169360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } }
I20260810 03:24:57.227309 11648 raft_consensus.cc:388] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:57.227506 11648 raft_consensus.cc:482] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:57.227703 11648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:57.230204 11648 raft_consensus.cc:504] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090259a4f4047e78dc206ce9e169360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } }
I20260810 03:24:57.230830 11648 leader_election.cc:302] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e090259a4f4047e78dc206ce9e169360; no voters:
I20260810 03:24:57.231654 11648 leader_election.cc:288] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:57.231828 11659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:57.233562 11659 raft_consensus.cc:686] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [term 1 LEADER]: Becoming Leader. State: Replica: e090259a4f4047e78dc206ce9e169360, State: Running, Role: LEADER
I20260810 03:24:57.234275 11659 consensus_queue.cc:237] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090259a4f4047e78dc206ce9e169360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } }
I20260810 03:24:57.234532 11648 sys_catalog.cc:564] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:57.237174 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e090259a4f4047e78dc206ce9e169360. Latest consensus state: current_term: 1 leader_uuid: "e090259a4f4047e78dc206ce9e169360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090259a4f4047e78dc206ce9e169360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } } }
I20260810 03:24:57.237709 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:57.238386 11660 sys_catalog.cc:455] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e090259a4f4047e78dc206ce9e169360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090259a4f4047e78dc206ce9e169360" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } } }
I20260810 03:24:57.238844 11660 sys_catalog.cc:458] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:57.250445 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:57.226008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:55903 (local address 127.0.0.1:38981)
0810 03:24:57.226248 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:57.226257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:57.226888 (+ 631us) server_negotiation.cc:407] Connection header received
0810 03:24:57.227113 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:57.227119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:57.227198 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:57.227314 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:57.229112 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:57.229983 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:57.231607 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:57.232529 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:57.232673 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:57.237416 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:57.237426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:57.237434 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:24:57.237509 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:57.243831 (+ 6322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:57.245692 (+ 1861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:57.245706 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:57.245712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:57.245956 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:24:57.246785 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:24:57.246791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:24:57.246794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:24:57.247381 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:57.249222 (+ 1841us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:57.249489 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 03:24:57.249788 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:24:57.252524 11664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:57.261658 11664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:57.274117 11664 catalog_manager.cc:1347] Generated new cluster ID: b581bc1d6c034400a9b369e53394e7dd
I20260810 03:24:57.274273 11664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:57.288286 11664 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:57.290527 11664 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:57.306116 11664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e090259a4f4047e78dc206ce9e169360: Generated new TSK 0
I20260810 03:24:57.307147 11664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:57.320075 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11576
I20260810 03:24:57.322016 11643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:24:57.420768 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 11576
W20260810 03:24:57.442867 11559 connection.cc:535] client connection to 127.0.0.1:38981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38981): Connection reset by peer (error 104)
2026-08-10T03:24:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:57 dist-test-slave-btw8 krb5kdc[11685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:57 dist-test-slave-btw8 krb5kdc[11685](info): set up 2 sockets
Aug 10 03:24:57 dist-test-slave-btw8 krb5kdc[11685](info): commencing operation
krb5kdc: starting...
W20260810 03:24:58.665064 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.005s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:24:58 dist-test-slave-btw8 krb5kdc[11685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:24:59.056399 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:24:59.454941 11700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:24:59.455269 11700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:24:59.455476 11700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:24:59.463462 11700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:24:59.463660 11700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:24:59.463828 11700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:24:59.463967 11700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:24:59.464049 11700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:24:59.475005 11700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:24:59.476583 11700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:24:59.477962 11700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:24:59.492005 11706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:59.492133 11709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:24:59.492727 11707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:24:59.493145 11700 server_base.cc:1034] running on GCE node
I20260810 03:24:59.494194 11700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:24:59.495991 11700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:24:59.497347 11700 hybrid_clock.cc:648] HybridClock initialized: now 1786332299497285 us; error 87 us; skew 500 ppm
I20260810 03:24:59.497844 11700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:24:59 dist-test-slave-btw8 krb5kdc[11685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:24:59.502457 11700 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:24:59.504503 11700 webserver.cc:460] Webserver started at http://127.0.0.1:40101/ using document root <none> and password file <none>
I20260810 03:24:59.505367 11700 fs_manager.cc:359] Metadata directory not provided
I20260810 03:24:59.505556 11700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:24:59.505921 11700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:24:59.509585 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "15d23758f55f4c16b8f11908c7d292d0"
format_stamp: "Formatted at 2026-08-10 03:24:59 on dist-test-slave-btw8"
server_key: "17b1d2cfc4e836f0b47e1f9df9162f3e"
server_key_iv: "0e559247f684c4bfe514d519cecca64a"
server_key_version: "encryptionkey@0"
I20260810 03:24:59.510526 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "15d23758f55f4c16b8f11908c7d292d0"
format_stamp: "Formatted at 2026-08-10 03:24:59 on dist-test-slave-btw8"
server_key: "17b1d2cfc4e836f0b47e1f9df9162f3e"
server_key_iv: "0e559247f684c4bfe514d519cecca64a"
server_key_version: "encryptionkey@0"
I20260810 03:24:59.517127 11700 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20260810 03:24:59.521615 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:24:59.523245 11700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:24:59.523545 11700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "15d23758f55f4c16b8f11908c7d292d0"
format_stamp: "Formatted at 2026-08-10 03:24:59 on dist-test-slave-btw8"
server_key: "17b1d2cfc4e836f0b47e1f9df9162f3e"
server_key_iv: "0e559247f684c4bfe514d519cecca64a"
server_key_version: "encryptionkey@0"
I20260810 03:24:59.523813 11700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:24:59.551118 11700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:24:59.556660 11700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:24:59.557026 11700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:24:59.579330 11700 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46239
I20260810 03:24:59.579427 11777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46239 every 8 connection(s)
I20260810 03:24:59.581069 11700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:24:59.585408 11778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:24:59.586745 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 11700
I20260810 03:24:59.586942 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:24:59.587565 20443 external_mini_cluster.cc:1428] Setting key 3d9bf8e5eec21cda9e5435b7d33c0514
I20260810 03:24:59.601289 11778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0: Bootstrap starting.
I20260810 03:24:59.603825 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:59.589708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:37085 (local address 127.0.0.1:46239)
0810 03:24:59.590691 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:59.590708 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:59.590753 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:24:59.592731 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:59.592763 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:59.593251 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:59.593697 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:59.594317 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:59.595985 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:59.598084 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:59.599055 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:59.599225 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:59.600859 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:59.600869 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:59.600885 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:59.601664 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:59.601769 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:59.601815 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:59.601975 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:59.602425 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 03:24:59.602696 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":455,"threads_started":1}
I20260810 03:24:59.605052 11778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0: Neither blocks nor log segments found. Creating new log.
I20260810 03:24:59.606675 11778 log.cc:824] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:24:59.608719 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:24:59.610390 11778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0: No bootstrap required, opened a new log
I20260810 03:24:59.621235 11778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d23758f55f4c16b8f11908c7d292d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 03:24:59.621853 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:24:59.614313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:38861 (local address 127.0.0.1:46239)
0810 03:24:59.614525 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 03:24:59.614532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:24:59.615037 (+ 505us) server_negotiation.cc:407] Connection header received
0810 03:24:59.615206 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:24:59.615212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:24:59.615305 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:24:59.615458 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:24:59.616076 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:59.617284 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:59.618676 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:24:59.619471 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:24:59.619627 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:24:59.620993 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:24:59.620999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:24:59.621003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:24:59.621029 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:24:59.621101 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:24:59.621124 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:24:59.621263 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:24:59.621440 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 03:24:59.621502 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:24:59.621862 11778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:24:59.622059 11778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d23758f55f4c16b8f11908c7d292d0, State: Initialized, Role: FOLLOWER
I20260810 03:24:59.622785 11778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d23758f55f4c16b8f11908c7d292d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 03:24:59.623072 11778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:24:59.623248 11778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:24:59.623478 11778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:24:59.625841 11778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d23758f55f4c16b8f11908c7d292d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 03:24:59.626407 11778 leader_election.cc:302] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15d23758f55f4c16b8f11908c7d292d0; no voters:
I20260810 03:24:59.627058 11778 leader_election.cc:288] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:24:59.627239 11790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:24:59.628104 11790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [term 1 LEADER]: Becoming Leader. State: Replica: 15d23758f55f4c16b8f11908c7d292d0, State: Running, Role: LEADER
I20260810 03:24:59.628849 11790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d23758f55f4c16b8f11908c7d292d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260810 03:24:59.629501 11778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:24:59.632903 11793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d23758f55f4c16b8f11908c7d292d0. Latest consensus state: current_term: 1 leader_uuid: "15d23758f55f4c16b8f11908c7d292d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d23758f55f4c16b8f11908c7d292d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } } }
I20260810 03:24:59.633378 11793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:59.634120 11791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15d23758f55f4c16b8f11908c7d292d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15d23758f55f4c16b8f11908c7d292d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } } }
I20260810 03:24:59.634477 11791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:24:59.634552 11798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:24:59.642038 11798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:24:59.653542 11798 catalog_manager.cc:1347] Generated new cluster ID: f9e7502a3dd446f2bec1fba2fccae92e
I20260810 03:24:59.653689 11798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:24:59.684698 11798 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:24:59.686396 11798 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:24:59.694444 11798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15d23758f55f4c16b8f11908c7d292d0: Generated new TSK 0
I20260810 03:24:59.695598 11798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:24:59.709277 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11700
W20260810 03:24:59.710678 11770 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:37085
W20260810 03:24:59.711707 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11700: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:24:59.711918 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 11700
W20260810 03:24:59.735955 11682 connection.cc:535] client connection to 127.0.0.1:46239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46239): Connection reset by peer (error 104)
2026-08-10T03:24:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:24:59 dist-test-slave-btw8 krb5kdc[11812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:24:59 dist-test-slave-btw8 krb5kdc[11812](info): set up 2 sockets
Aug 10 03:24:59 dist-test-slave-btw8 krb5kdc[11812](info): commencing operation
krb5kdc: starting...
W20260810 03:25:00.969069 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.003s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:01 dist-test-slave-btw8 krb5kdc[11812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:25:01.360735 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:01.739341 11826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:01.739775 11826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:01.739956 11826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:01.747805 11826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:01.748016 11826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:01.748142 11826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:01.748255 11826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:01.748333 11826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:01.759572 11826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:01.761106 11826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:01.762580 11826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:01.777318 11831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:01.777371 11832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:01.777415 11834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:01.779629 11826 server_base.cc:1034] running on GCE node
I20260810 03:25:01.780665 11826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:01.782382 11826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:01.783654 11826 hybrid_clock.cc:648] HybridClock initialized: now 1786332301783572 us; error 66 us; skew 500 ppm
I20260810 03:25:01.784080 11826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:01 dist-test-slave-btw8 krb5kdc[11812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:01.788395 11826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:25:01.790446 11826 webserver.cc:460] Webserver started at http://127.0.0.1:44597/ using document root <none> and password file <none>
I20260810 03:25:01.791288 11826 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:01.791514 11826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:01.791834 11826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:01.795636 11826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "99433903c8b445ea865052d20712385c"
format_stamp: "Formatted at 2026-08-10 03:25:01 on dist-test-slave-btw8"
server_key: "97335415017340f093b373fad01051a4"
server_key_iv: "a8d0cb6c7f1149a2d7e94175c758ddae"
server_key_version: "encryptionkey@0"
I20260810 03:25:01.796578 11826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "99433903c8b445ea865052d20712385c"
format_stamp: "Formatted at 2026-08-10 03:25:01 on dist-test-slave-btw8"
server_key: "97335415017340f093b373fad01051a4"
server_key_iv: "a8d0cb6c7f1149a2d7e94175c758ddae"
server_key_version: "encryptionkey@0"
I20260810 03:25:01.803951 11826 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 03:25:01.808542 11842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:01.810451 11826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:25:01.810732 11826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "99433903c8b445ea865052d20712385c"
format_stamp: "Formatted at 2026-08-10 03:25:01 on dist-test-slave-btw8"
server_key: "97335415017340f093b373fad01051a4"
server_key_iv: "a8d0cb6c7f1149a2d7e94175c758ddae"
server_key_version: "encryptionkey@0"
I20260810 03:25:01.810988 11826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:01.818061 11826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:01.824501 11826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:01.824944 11826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:01.850565 11826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37137
I20260810 03:25:01.850636 11896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37137 every 8 connection(s)
I20260810 03:25:01.852320 11826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:01.857134 11897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:01.862394 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 11826
I20260810 03:25:01.862735 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:01.863212 20443 external_mini_cluster.cc:1428] Setting key bd197e3f2b596adab99959d0fa3a7b8e
I20260810 03:25:01.870098 11897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c: Bootstrap starting.
I20260810 03:25:01.872354 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:01.865312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:46197 (local address 127.0.0.1:37137)
0810 03:25:01.866230 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:01.866243 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:01.866290 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:25:01.868444 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:01.868497 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:01.868914 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:01.869358 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:01.870547 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:01.870554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:01.870568 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:01.870584 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:01.870649 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:01.870671 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:01.870931 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:01.871075 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 03:25:01.871303 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":373,"threads_started":1}
I20260810 03:25:01.874318 11897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:01.875703 11897 log.cc:824] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:01.876713 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:01.878886 11897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c: No bootstrap required, opened a new log
I20260810 03:25:01.884428 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:01.881439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:47155 (local address 127.0.0.1:37137)
0810 03:25:01.881620 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:01.881628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:01.882102 (+ 474us) server_negotiation.cc:407] Connection header received
0810 03:25:01.882312 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:01.882317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:01.882378 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:01.882501 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:01.883624 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:01.883630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:01.883636 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:01.883665 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:01.883728 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:01.883745 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:01.883899 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:01.884009 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 03:25:01.884094 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:25:01.888420 11897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99433903c8b445ea865052d20712385c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } }
I20260810 03:25:01.889047 11897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:01.889210 11897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99433903c8b445ea865052d20712385c, State: Initialized, Role: FOLLOWER
I20260810 03:25:01.889981 11897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99433903c8b445ea865052d20712385c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } }
I20260810 03:25:01.890213 11897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:01.890323 11897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:01.890460 11897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:01.892900 11897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99433903c8b445ea865052d20712385c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } }
I20260810 03:25:01.893466 11897 leader_election.cc:302] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99433903c8b445ea865052d20712385c; no voters:
I20260810 03:25:01.894129 11897 leader_election.cc:288] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:01.894289 11907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:01.895632 11907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [term 1 LEADER]: Becoming Leader. State: Replica: 99433903c8b445ea865052d20712385c, State: Running, Role: LEADER
I20260810 03:25:01.896235 11907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99433903c8b445ea865052d20712385c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } }
I20260810 03:25:01.896534 11897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:01.899415 11909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99433903c8b445ea865052d20712385c. Latest consensus state: current_term: 1 leader_uuid: "99433903c8b445ea865052d20712385c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99433903c8b445ea865052d20712385c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } } }
I20260810 03:25:01.899325 11908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99433903c8b445ea865052d20712385c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99433903c8b445ea865052d20712385c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } } }
I20260810 03:25:01.899983 11909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:01.900053 11908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:01.901005 11917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:01.916277 11917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:01.927747 11917 catalog_manager.cc:1347] Generated new cluster ID: 21292e1eabea4c82ab0983d6f8f6272a
I20260810 03:25:01.927917 11917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:01.953881 11917 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:01.954510 11917 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:01.967316 11917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99433903c8b445ea865052d20712385c: Generated new TSK 0
I20260810 03:25:01.968658 11917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:25:01.986644 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11826
W20260810 03:25:01.987911 11892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:46197
W20260810 03:25:01.988744 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11826: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:01.988909 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 11826
2026-08-10T03:25:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:02 dist-test-slave-btw8 krb5kdc[11931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:02 dist-test-slave-btw8 krb5kdc[11931](info): set up 2 sockets
Aug 10 03:25:02 dist-test-slave-btw8 krb5kdc[11931](info): commencing operation
krb5kdc: starting...
W20260810 03:25:03.222857 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.005s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:03 dist-test-slave-btw8 krb5kdc[11931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:25:03.610512 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40841
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:40841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:03.976161 11944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:03.976485 11944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:03.976661 11944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:03.984447 11944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:03.984593 11944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:03.984642 11944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:03.984704 11944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:03.984751 11944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:03.995344 11944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:40841
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:40841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:03.996816 11944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:03.998190 11944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:04.011410 11949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:04.011471 11950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:04.011535 11952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:04.012374 11944 server_base.cc:1034] running on GCE node
I20260810 03:25:04.013290 11944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:04.014971 11944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:04.016224 11944 hybrid_clock.cc:648] HybridClock initialized: now 1786332304016168 us; error 61 us; skew 500 ppm
I20260810 03:25:04.016675 11944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:04 dist-test-slave-btw8 krb5kdc[11931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332304, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:04.020177 11944 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:25:04.021859 11944 webserver.cc:460] Webserver started at http://127.19.246.254:33241/ using document root <none> and password file <none>
I20260810 03:25:04.022567 11944 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:04.022677 11944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:04.022950 11944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:04.026340 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "059a8d685ab644f9a13eb1b4624a6f74"
format_stamp: "Formatted at 2026-08-10 03:25:04 on dist-test-slave-btw8"
server_key: "7243783c9719024d4fd6a3e9c57342a9"
server_key_iv: "f972140f77bcb522cbe4e5d615f16107"
server_key_version: "encryptionkey@0"
I20260810 03:25:04.027242 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "059a8d685ab644f9a13eb1b4624a6f74"
format_stamp: "Formatted at 2026-08-10 03:25:04 on dist-test-slave-btw8"
server_key: "7243783c9719024d4fd6a3e9c57342a9"
server_key_iv: "f972140f77bcb522cbe4e5d615f16107"
server_key_version: "encryptionkey@0"
I20260810 03:25:04.033591 11944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:25:04.037891 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:04.039397 11944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:25:04.039624 11944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "059a8d685ab644f9a13eb1b4624a6f74"
format_stamp: "Formatted at 2026-08-10 03:25:04 on dist-test-slave-btw8"
server_key: "7243783c9719024d4fd6a3e9c57342a9"
server_key_iv: "f972140f77bcb522cbe4e5d615f16107"
server_key_version: "encryptionkey@0"
I20260810 03:25:04.039863 11944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:04.080977 11944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:04.086213 11944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:04.086560 11944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:04.108083 11944 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:40841
I20260810 03:25:04.108167 12014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:40841 every 8 connection(s)
I20260810 03:25:04.109688 11944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:04.113940 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:04.118865 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 11944
I20260810 03:25:04.119155 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:04.119740 20443 external_mini_cluster.cc:1428] Setting key 58695216bd33286765fc89c3ef596883
I20260810 03:25:04.127560 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74: Bootstrap starting.
I20260810 03:25:04.131722 12015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:04.133191 12015 log.cc:824] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:04.136576 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74: No bootstrap required, opened a new log
Aug 10 03:25:04 dist-test-slave-btw8 krb5kdc[11931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:25:04.148259 12015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a8d685ab644f9a13eb1b4624a6f74" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40841 } }
I20260810 03:25:04.148801 12015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:04.148916 12015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a8d685ab644f9a13eb1b4624a6f74, State: Initialized, Role: FOLLOWER
I20260810 03:25:04.149106 12020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:04.121923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35795 (local address 127.19.246.254:40841)
0810 03:25:04.122803 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:04.122819 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:04.122869 (+ 50us) server_negotiation.cc:407] Connection header received
0810 03:25:04.125528 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:04.125563 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:04.126156 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:04.126649 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:04.128366 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:04.130531 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:04.132120 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:04.133270 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:04.133466 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:04.138156 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:04.138170 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:04.138191 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:04.138241 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:04.143001 (+ 4760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:04.144439 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:04.144450 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:04.144461 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:04.144701 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:04.145326 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:04.145337 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:04.145340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:04.146208 (+ 868us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:04.146572 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:04.147153 (+ 581us) server_negotiation.cc:299] Negotiation successful
0810 03:25:04.147573 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":341,"threads_started":1}
I20260810 03:25:04.149679 12015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a8d685ab644f9a13eb1b4624a6f74" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40841 } }
I20260810 03:25:04.150003 12015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:04.150156 12015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:04.150348 12015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:04.153479 12015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a8d685ab644f9a13eb1b4624a6f74" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40841 } }
I20260810 03:25:04.153605 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:04.154026 12015 leader_election.cc:302] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 059a8d685ab644f9a13eb1b4624a6f74; no voters:
I20260810 03:25:04.154734 12015 leader_election.cc:288] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:04.154843 12022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:04.157572 12015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:04.162007 12022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [term 1 LEADER]: Becoming Leader. State: Replica: 059a8d685ab644f9a13eb1b4624a6f74, State: Running, Role: LEADER
I20260810 03:25:04.162942 12022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a8d685ab644f9a13eb1b4624a6f74" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40841 } }
I20260810 03:25:04.166822 12023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "059a8d685ab644f9a13eb1b4624a6f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a8d685ab644f9a13eb1b4624a6f74" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40841 } } }
I20260810 03:25:04.167568 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059a8d685ab644f9a13eb1b4624a6f74. Latest consensus state: current_term: 1 leader_uuid: "059a8d685ab644f9a13eb1b4624a6f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a8d685ab644f9a13eb1b4624a6f74" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 40841 } } }
I20260810 03:25:04.167910 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:04.167371 12023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:04.169157 12038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:04.181646 12038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:04.194814 12038 catalog_manager.cc:1347] Generated new cluster ID: 9365b9568846452395bfd0b4e8564892
I20260810 03:25:04.194998 12038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:04.195406 12020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:04.166220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45979 (local address 127.19.246.254:40841)
0810 03:25:04.166396 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:04.166405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:04.173389 (+ 6984us) server_negotiation.cc:407] Connection header received
0810 03:25:04.173782 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:04.173792 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:04.173908 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:04.174010 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:04.176902 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:04.178284 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:04.180225 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:04.180890 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:04.181003 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:04.185173 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:04.185182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:04.185187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:04.185247 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:04.189656 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:04.191322 (+ 1666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:04.191326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:04.191329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:04.191587 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:04.192291 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:04.192302 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:04.192307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:04.192795 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:04.193107 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:04.194702 (+ 1595us) server_negotiation.cc:299] Negotiation successful
0810 03:25:04.194989 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 03:25:04.213287 12038 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:04.215684 12038 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:04.226974 12038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 059a8d685ab644f9a13eb1b4624a6f74: Generated new TSK 0
I20260810 03:25:04.228281 12038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:25:04.246629 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 11944
I20260810 03:25:04.248674 12010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:04.347676 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 11944
W20260810 03:25:04.368726 11929 connection.cc:535] client connection to 127.19.246.254:40841 recv error: Network error: failed to read from TLS socket (remote: 127.19.246.254:40841): Connection reset by peer (error 104)
2026-08-10T03:25:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:04 dist-test-slave-btw8 krb5kdc[12052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:04 dist-test-slave-btw8 krb5kdc[12052](info): set up 2 sockets
Aug 10 03:25:04 dist-test-slave-btw8 krb5kdc[12052](info): commencing operation
krb5kdc: starting...
W20260810 03:25:05.639145 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:05 dist-test-slave-btw8 krb5kdc[12052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:25:06.027221 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:38977
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:38977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:06.422361 12065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:06.422704 12065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:06.422894 12065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:06.430622 12065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:06.430754 12065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:06.430847 12065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:06.430962 12065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:06.431025 12065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:06.441663 12065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:38977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:38977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:06.443146 12065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:06.444592 12065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:06.458357 12071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:06.458417 12072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:06.458437 12074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:06.459425 12065 server_base.cc:1034] running on GCE node
I20260810 03:25:06.460397 12065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:06.462015 12065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:06.463279 12065 hybrid_clock.cc:648] HybridClock initialized: now 1786332306463231 us; error 44 us; skew 500 ppm
I20260810 03:25:06.463717 12065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:06 dist-test-slave-btw8 krb5kdc[12052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332306, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:06.473380 12065 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:25:06.475167 12065 webserver.cc:460] Webserver started at http://127.19.246.254:41745/ using document root <none> and password file <none>
I20260810 03:25:06.476002 12065 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:06.476169 12065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:06.476495 12065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:06.479758 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "6c634598ef12445e9aac5f7b3dd7d4cc"
format_stamp: "Formatted at 2026-08-10 03:25:06 on dist-test-slave-btw8"
server_key: "ca76d4a7726c0973aaafba72114363f2"
server_key_iv: "cf01e9ab41c47105065f6aae688e5f03"
server_key_version: "encryptionkey@0"
I20260810 03:25:06.480664 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "6c634598ef12445e9aac5f7b3dd7d4cc"
format_stamp: "Formatted at 2026-08-10 03:25:06 on dist-test-slave-btw8"
server_key: "ca76d4a7726c0973aaafba72114363f2"
server_key_iv: "cf01e9ab41c47105065f6aae688e5f03"
server_key_version: "encryptionkey@0"
I20260810 03:25:06.486526 12065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 03:25:06.490522 12081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:06.492223 12065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:25:06.492451 12065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "6c634598ef12445e9aac5f7b3dd7d4cc"
format_stamp: "Formatted at 2026-08-10 03:25:06 on dist-test-slave-btw8"
server_key: "ca76d4a7726c0973aaafba72114363f2"
server_key_iv: "cf01e9ab41c47105065f6aae688e5f03"
server_key_version: "encryptionkey@0"
I20260810 03:25:06.492682 12065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:06.510154 12065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:06.516063 12065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:06.516458 12065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:06.538349 12065 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:38977
I20260810 03:25:06.538362 12139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:38977 every 8 connection(s)
I20260810 03:25:06.540093 12065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:06.544385 12140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:06.547837 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 12065
I20260810 03:25:06.548067 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:06.548612 20443 external_mini_cluster.cc:1428] Setting key e05cfe8d58462359808590583b6949d8
I20260810 03:25:06.557998 12140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc: Bootstrap starting.
I20260810 03:25:06.562088 12140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:06.563737 12140 log.cc:824] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:06.564688 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:06.550768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35381 (local address 127.19.246.254:38977)
0810 03:25:06.551714 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:06.551727 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:06.551773 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:25:06.553617 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:06.553644 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:06.554148 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:06.554527 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:06.555243 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:06.556917 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:06.558946 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:06.559761 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:06.559924 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:06.561350 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:06.561358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:06.561373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:06.561941 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:06.562038 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:06.562070 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:06.562243 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:06.562805 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 03:25:06.563042 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":477,"threads_started":1}
I20260810 03:25:06.567140 12140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc: No bootstrap required, opened a new log
I20260810 03:25:06.569336 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:06.576249 12140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c634598ef12445e9aac5f7b3dd7d4cc" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38977 } }
I20260810 03:25:06.576745 12140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:06.576874 12140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c634598ef12445e9aac5f7b3dd7d4cc, State: Initialized, Role: FOLLOWER
I20260810 03:25:06.577557 12140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c634598ef12445e9aac5f7b3dd7d4cc" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38977 } }
I20260810 03:25:06.577883 12140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:06.578038 12140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:06.578270 12140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:06.581619 12140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c634598ef12445e9aac5f7b3dd7d4cc" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38977 } }
I20260810 03:25:06.582541 12140 leader_election.cc:302] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c634598ef12445e9aac5f7b3dd7d4cc; no voters:
I20260810 03:25:06.582994 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:06.574155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48301 (local address 127.19.246.254:38977)
0810 03:25:06.574412 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:06.574423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:06.574999 (+ 576us) server_negotiation.cc:407] Connection header received
0810 03:25:06.575307 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:06.575312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:06.575462 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:06.575629 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:06.576316 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:06.577363 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:06.579449 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:06.580105 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:06.580233 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:06.581727 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:06.581733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:06.581736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:06.581759 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:06.581872 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:06.581903 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:06.582018 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:06.582410 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 03:25:06.582500 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:25:06.583803 12140 leader_election.cc:288] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:06.583882 12150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:06.586467 12140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:06.587416 12150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [term 1 LEADER]: Becoming Leader. State: Replica: 6c634598ef12445e9aac5f7b3dd7d4cc, State: Running, Role: LEADER
I20260810 03:25:06.588060 12150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c634598ef12445e9aac5f7b3dd7d4cc" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38977 } }
I20260810 03:25:06.591996 12151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c634598ef12445e9aac5f7b3dd7d4cc. Latest consensus state: current_term: 1 leader_uuid: "6c634598ef12445e9aac5f7b3dd7d4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c634598ef12445e9aac5f7b3dd7d4cc" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38977 } } }
I20260810 03:25:06.591918 12152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c634598ef12445e9aac5f7b3dd7d4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c634598ef12445e9aac5f7b3dd7d4cc" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38977 } } }
I20260810 03:25:06.592456 12151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:06.592474 12152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:06.593719 12160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:06.599278 12160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:06.609068 12160 catalog_manager.cc:1347] Generated new cluster ID: 036c81d06817412cb83118d44b4c3013
I20260810 03:25:06.609226 12160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:06.635814 12160 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:06.637868 12160 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:06.655627 12160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c634598ef12445e9aac5f7b3dd7d4cc: Generated new TSK 0
I20260810 03:25:06.656781 12160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:25:06.670933 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12065
W20260810 03:25:06.672358 12135 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35381
W20260810 03:25:06.673327 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12065: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:06.673484 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 12065
W20260810 03:25:06.691331 12049 connection.cc:535] client connection to 127.19.246.254:38977 recv error: Network error: failed to read from TLS socket (remote: 127.19.246.254:38977): Connection reset by peer (error 104)
2026-08-10T03:25:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:06 dist-test-slave-btw8 krb5kdc[12172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:06 dist-test-slave-btw8 krb5kdc[12172](info): set up 2 sockets
Aug 10 03:25:06 dist-test-slave-btw8 krb5kdc[12172](info): commencing operation
krb5kdc: starting...
W20260810 03:25:07.919587 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.010s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:08 dist-test-slave-btw8 krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:25:08.296110 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:44805
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:08.661927 12185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:08.662274 12185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:08.662431 12185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:08.669922 12185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:08.670053 12185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:08.670166 12185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:08.670269 12185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:08.670343 12185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:08.680859 12185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:44805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:08.682256 12185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:08.683725 12185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:08.696076 12193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:08.696020 12191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:08.696041 12190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:08.696940 12185 server_base.cc:1034] running on GCE node
I20260810 03:25:08.697715 12185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:08.699193 12185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:08.700441 12185 hybrid_clock.cc:648] HybridClock initialized: now 1786332308700421 us; error 51 us; skew 500 ppm
I20260810 03:25:08.700860 12185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:08 dist-test-slave-btw8 krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332308, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:08.704358 12185 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:25:08.706501 12185 webserver.cc:460] Webserver started at http://127.19.246.254:40661/ using document root <none> and password file <none>
I20260810 03:25:08.707229 12185 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:08.707404 12185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:08.707723 12185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:08.710922 12185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "ebe5cd888c494789b7455f4b08e45b81"
format_stamp: "Formatted at 2026-08-10 03:25:08 on dist-test-slave-btw8"
server_key: "753a9935976e03d8a16627b55eb87a88"
server_key_iv: "7c560b75908ca9e5db5f661dab2671c4"
server_key_version: "encryptionkey@0"
I20260810 03:25:08.711810 12185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "ebe5cd888c494789b7455f4b08e45b81"
format_stamp: "Formatted at 2026-08-10 03:25:08 on dist-test-slave-btw8"
server_key: "753a9935976e03d8a16627b55eb87a88"
server_key_iv: "7c560b75908ca9e5db5f661dab2671c4"
server_key_version: "encryptionkey@0"
I20260810 03:25:08.717753 12185 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 03:25:08.721596 12200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:08.723151 12185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:25:08.723356 12185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "ebe5cd888c494789b7455f4b08e45b81"
format_stamp: "Formatted at 2026-08-10 03:25:08 on dist-test-slave-btw8"
server_key: "753a9935976e03d8a16627b55eb87a88"
server_key_iv: "7c560b75908ca9e5db5f661dab2671c4"
server_key_version: "encryptionkey@0"
I20260810 03:25:08.723584 12185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:08.729957 12185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:08.734776 12185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:08.735152 12185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:08.755856 12185 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:44805
I20260810 03:25:08.755877 12256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:44805 every 8 connection(s)
I20260810 03:25:08.757474 12185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:08.761866 12257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:08.763228 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 12185
I20260810 03:25:08.763458 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:08.764146 20443 external_mini_cluster.cc:1428] Setting key 5f10b31fbd4429f28b4c0d9f749250a2
I20260810 03:25:08.773190 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:08.766412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60073 (local address 127.19.246.254:44805)
0810 03:25:08.767441 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:08.767459 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:08.767507 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:25:08.769464 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:08.769499 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:08.769902 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:08.770340 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:08.771495 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:08.771502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:08.771516 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:08.771532 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:08.771607 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:08.771629 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:08.771881 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:08.772021 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 03:25:08.772229 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":396,"threads_started":1}
I20260810 03:25:08.777045 12257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81: Bootstrap starting.
I20260810 03:25:08.777206 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:08.780478 12257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:08.781935 12257 log.cc:824] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:08.785197 12257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81: No bootstrap required, opened a new log
I20260810 03:25:08.789165 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:08.781887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41283 (local address 127.19.246.254:44805)
0810 03:25:08.782130 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:08.782137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:08.782325 (+ 188us) server_negotiation.cc:407] Connection header received
0810 03:25:08.782531 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:08.782545 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:08.782598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:08.782710 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:08.788371 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:08.788380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:08.788383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:08.788404 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:08.788452 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:08.788469 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:08.788620 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:08.788789 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 03:25:08.788880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:25:08.793761 12257 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe5cd888c494789b7455f4b08e45b81" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44805 } }
I20260810 03:25:08.794278 12257 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:08.794391 12257 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe5cd888c494789b7455f4b08e45b81, State: Initialized, Role: FOLLOWER
I20260810 03:25:08.795027 12257 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe5cd888c494789b7455f4b08e45b81" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44805 } }
I20260810 03:25:08.795279 12257 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:08.795463 12257 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:08.795634 12257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:08.798147 12257 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe5cd888c494789b7455f4b08e45b81" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44805 } }
I20260810 03:25:08.798727 12257 leader_election.cc:302] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe5cd888c494789b7455f4b08e45b81; no voters:
I20260810 03:25:08.799413 12257 leader_election.cc:288] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:08.799592 12268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:08.800359 12268 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [term 1 LEADER]: Becoming Leader. State: Replica: ebe5cd888c494789b7455f4b08e45b81, State: Running, Role: LEADER
I20260810 03:25:08.800997 12268 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe5cd888c494789b7455f4b08e45b81" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44805 } }
I20260810 03:25:08.801719 12257 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:08.804399 12270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe5cd888c494789b7455f4b08e45b81. Latest consensus state: current_term: 1 leader_uuid: "ebe5cd888c494789b7455f4b08e45b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe5cd888c494789b7455f4b08e45b81" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44805 } } }
I20260810 03:25:08.804375 12269 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebe5cd888c494789b7455f4b08e45b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe5cd888c494789b7455f4b08e45b81" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44805 } } }
I20260810 03:25:08.804908 12270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:08.804916 12269 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:08.807168 12274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:08.813057 12274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:08.823943 12274 catalog_manager.cc:1347] Generated new cluster ID: 27aa9da90f114cc995ea42c707b531ae
I20260810 03:25:08.824124 12274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:08.840102 12274 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:08.840703 12274 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:08.850550 12274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebe5cd888c494789b7455f4b08e45b81: Generated new TSK 0
I20260810 03:25:08.851804 12274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:25:08.871281 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12185
W20260810 03:25:08.872812 12252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60073
W20260810 03:25:08.873622 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:08.873788 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 12185
2026-08-10T03:25:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:09 dist-test-slave-btw8 krb5kdc[12290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:09 dist-test-slave-btw8 krb5kdc[12290](info): set up 2 sockets
Aug 10 03:25:09 dist-test-slave-btw8 krb5kdc[12290](info): commencing operation
krb5kdc: starting...
W20260810 03:25:10.135010 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:10 dist-test-slave-btw8 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:25:10.509241 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:44883
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:42017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:44883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:10.874425 12305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:10.874781 12305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:10.874881 12305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:10.882687 12305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:10.882829 12305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:10.882877 12305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:10.882933 12305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:10.882979 12305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:10.893550 12305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:42017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:44883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:44883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:10.894953 12305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:10.896387 12305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:10.909611 12313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:10.909770 12310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:10.909642 12311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:10.910708 12305 server_base.cc:1034] running on GCE node
I20260810 03:25:10.911689 12305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:10.913327 12305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:10.914548 12305 hybrid_clock.cc:648] HybridClock initialized: now 1786332310914508 us; error 39 us; skew 500 ppm
I20260810 03:25:10.914952 12305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:10 dist-test-slave-btw8 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332310, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:10.918752 12305 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:25:10.920504 12305 webserver.cc:460] Webserver started at http://127.19.246.254:36485/ using document root <none> and password file <none>
I20260810 03:25:10.921300 12305 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:10.921452 12305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:10.921768 12305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:10.925158 12305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "f65ab00e13b84d02927b89b0cb8140c8"
format_stamp: "Formatted at 2026-08-10 03:25:10 on dist-test-slave-btw8"
server_key: "3bb1427032c4f6b1fb86a3f04622637c"
server_key_iv: "5e1c5ad68c18fe1a5473b4cadf6b18e8"
server_key_version: "encryptionkey@0"
I20260810 03:25:10.925999 12305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "f65ab00e13b84d02927b89b0cb8140c8"
format_stamp: "Formatted at 2026-08-10 03:25:10 on dist-test-slave-btw8"
server_key: "3bb1427032c4f6b1fb86a3f04622637c"
server_key_iv: "5e1c5ad68c18fe1a5473b4cadf6b18e8"
server_key_version: "encryptionkey@0"
I20260810 03:25:10.932443 12305 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 03:25:10.936889 12320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:10.938468 12305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:25:10.938657 12305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "f65ab00e13b84d02927b89b0cb8140c8"
format_stamp: "Formatted at 2026-08-10 03:25:10 on dist-test-slave-btw8"
server_key: "3bb1427032c4f6b1fb86a3f04622637c"
server_key_iv: "5e1c5ad68c18fe1a5473b4cadf6b18e8"
server_key_version: "encryptionkey@0"
I20260810 03:25:10.938858 12305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:10.967133 12305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:10.972636 12305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:10.972918 12305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:10.994001 12305 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:44883
I20260810 03:25:10.994046 12378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:44883 every 8 connection(s)
I20260810 03:25:10.995635 12305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:10.998154 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 12305
I20260810 03:25:10.998355 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:10.998831 20443 external_mini_cluster.cc:1428] Setting key 119b685a18eedc9bd1ac89da6c084956
I20260810 03:25:11.000658 12380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:25:11 dist-test-slave-btw8 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:25:11.017402 12380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8: Bootstrap starting.
I20260810 03:25:11.021502 12380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:11.023059 12380 log.cc:824] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:11.026466 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:11.000910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33291 (local address 127.19.246.254:44883)
0810 03:25:11.001807 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:11.001820 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:11.001861 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:25:11.003586 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:11.003611 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:11.004054 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:11.004419 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:11.006701 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:11.008401 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:11.010572 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:11.011344 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:11.011672 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:11.017161 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:11.017173 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:11.017190 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:11.017250 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:11.020948 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:11.021798 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:11.021808 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:11.021818 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:11.022098 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:11.022606 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:11.022612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:11.022616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:11.023524 (+ 908us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:11.023951 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:11.024459 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 03:25:11.024937 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":365,"threads_started":1}
I20260810 03:25:11.027206 12380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8: No bootstrap required, opened a new log
I20260810 03:25:11.030577 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:11.038754 12380 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65ab00e13b84d02927b89b0cb8140c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44883 } }
I20260810 03:25:11.039223 12380 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:11.039423 12380 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65ab00e13b84d02927b89b0cb8140c8, State: Initialized, Role: FOLLOWER
I20260810 03:25:11.040122 12380 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65ab00e13b84d02927b89b0cb8140c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44883 } }
I20260810 03:25:11.040390 12380 raft_consensus.cc:388] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:11.040573 12380 raft_consensus.cc:482] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:11.040779 12380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:11.043975 12380 raft_consensus.cc:504] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65ab00e13b84d02927b89b0cb8140c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44883 } }
I20260810 03:25:11.044597 12380 leader_election.cc:302] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65ab00e13b84d02927b89b0cb8140c8; no voters:
I20260810 03:25:11.045396 12380 leader_election.cc:288] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:11.045535 12391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:11.048913 12380 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:11.053211 12391 raft_consensus.cc:686] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [term 1 LEADER]: Becoming Leader. State: Replica: f65ab00e13b84d02927b89b0cb8140c8, State: Running, Role: LEADER
I20260810 03:25:11.053556 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:11.036659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41721 (local address 127.19.246.254:44883)
0810 03:25:11.036891 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:11.036898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:11.037242 (+ 344us) server_negotiation.cc:407] Connection header received
0810 03:25:11.037395 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:11.037401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:11.037493 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:11.037631 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:11.039350 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:11.040348 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:11.041735 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:11.042273 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:11.042445 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:11.045459 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:11.045464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:11.045467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:11.045536 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:11.049020 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:11.049875 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:11.049881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:11.049885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:11.050038 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:11.050460 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:11.050465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:11.050469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:11.050848 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:11.052563 (+ 1715us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:11.052822 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 03:25:11.053094 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:25:11.054155 12391 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65ab00e13b84d02927b89b0cb8140c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44883 } }
I20260810 03:25:11.058081 12393 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65ab00e13b84d02927b89b0cb8140c8. Latest consensus state: current_term: 1 leader_uuid: "f65ab00e13b84d02927b89b0cb8140c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65ab00e13b84d02927b89b0cb8140c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44883 } } }
I20260810 03:25:11.058581 12393 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:11.059337 12392 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65ab00e13b84d02927b89b0cb8140c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65ab00e13b84d02927b89b0cb8140c8" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 44883 } } }
I20260810 03:25:11.059758 12392 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:11.062430 12403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:11.069429 12403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:11.080243 12403 catalog_manager.cc:1347] Generated new cluster ID: 6d0ca8670577483fa77399fe0d3a5077
I20260810 03:25:11.080397 12403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:11.093276 12403 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:11.095575 12403 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:11.105316 12403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f65ab00e13b84d02927b89b0cb8140c8: Generated new TSK 0
I20260810 03:25:11.106565 12403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:25:11.118435 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12305
I20260810 03:25:11.120190 12373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:11.217329 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 12305
2026-08-10T03:25:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:11 dist-test-slave-btw8 krb5kdc[12415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:11 dist-test-slave-btw8 krb5kdc[12415](info): set up 2 sockets
Aug 10 03:25:11 dist-test-slave-btw8 krb5kdc[12415](info): commencing operation
krb5kdc: starting...
W20260810 03:25:12.482769 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.001s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:12 dist-test-slave-btw8 krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:25:12.864670 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:33839
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:33839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:13.234745 12430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:13.235184 12430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:13.235381 12430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:13.243065 12430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:13.243273 12430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:13.243414 12430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:13.243533 12430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:13.243637 12430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:13.254340 12430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:33839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:33839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:13.255884 12430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:13.257262 12430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:13.270124 12436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:13.270179 12437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:13.270190 12439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:13.271425 12430 server_base.cc:1034] running on GCE node
I20260810 03:25:13.272339 12430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:13.273984 12430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:13.275264 12430 hybrid_clock.cc:648] HybridClock initialized: now 1786332313275213 us; error 47 us; skew 500 ppm
I20260810 03:25:13.275794 12430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:13 dist-test-slave-btw8 krb5kdc[12415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332313, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:13.283726 12430 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:25:13.285327 12430 webserver.cc:460] Webserver started at http://127.19.246.254:34673/ using document root <none> and password file <none>
I20260810 03:25:13.286062 12430 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:13.286172 12430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:13.286441 12430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:13.290124 12430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "2cac5ac36d7c414da3114351ed2f17ac"
format_stamp: "Formatted at 2026-08-10 03:25:13 on dist-test-slave-btw8"
server_key: "80f582788eb5b8785c0fbfb3f0ff004a"
server_key_iv: "cc0494ccf603725ebd07f1ce2dac593b"
server_key_version: "encryptionkey@0"
I20260810 03:25:13.290953 12430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "2cac5ac36d7c414da3114351ed2f17ac"
format_stamp: "Formatted at 2026-08-10 03:25:13 on dist-test-slave-btw8"
server_key: "80f582788eb5b8785c0fbfb3f0ff004a"
server_key_iv: "cc0494ccf603725ebd07f1ce2dac593b"
server_key_version: "encryptionkey@0"
I20260810 03:25:13.296828 12430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 03:25:13.300563 12446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:13.302129 12430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:25:13.302358 12430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "2cac5ac36d7c414da3114351ed2f17ac"
format_stamp: "Formatted at 2026-08-10 03:25:13 on dist-test-slave-btw8"
server_key: "80f582788eb5b8785c0fbfb3f0ff004a"
server_key_iv: "cc0494ccf603725ebd07f1ce2dac593b"
server_key_version: "encryptionkey@0"
I20260810 03:25:13.302564 12430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:13.329581 12430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:13.334805 12430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:13.335177 12430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:13.356869 12430 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:33839
I20260810 03:25:13.356812 12499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:33839 every 8 connection(s)
I20260810 03:25:13.359053 12430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:13.362701 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 12430
I20260810 03:25:13.362908 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:13.363539 20443 external_mini_cluster.cc:1428] Setting key aadfa852a49f925276259599dad52a60
I20260810 03:25:13.363605 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:13.378324 12500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac: Bootstrap starting.
I20260810 03:25:13.378423 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:13.365587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36783 (local address 127.19.246.254:33839)
0810 03:25:13.366491 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:13.366506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:13.366550 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:25:13.368573 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:13.368599 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:13.369067 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:13.369503 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:13.370126 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:13.371613 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:13.373500 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:13.374249 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:13.374413 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:13.375899 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:13.375907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:13.375925 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:13.376585 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:13.376670 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:13.376709 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:13.376883 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:13.377074 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 03:25:13.377376 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":427,"threads_started":1}
I20260810 03:25:13.382002 12500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:13.382679 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:13.383606 12500 log.cc:824] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:13.386847 12500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac: No bootstrap required, opened a new log
I20260810 03:25:13.395816 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:13.388203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57227 (local address 127.19.246.254:33839)
0810 03:25:13.388455 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:13.388462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:13.388866 (+ 404us) server_negotiation.cc:407] Connection header received
0810 03:25:13.389122 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:13.389129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:13.389202 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:13.389309 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:13.389946 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:13.390803 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:13.392837 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:13.393367 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:13.393488 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:13.394812 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:13.394817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:13.394820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:13.394843 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:13.394888 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:13.394909 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:13.395006 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:13.395212 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 03:25:13.395312 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 03:25:13.397449 12500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac5ac36d7c414da3114351ed2f17ac" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33839 } }
I20260810 03:25:13.397897 12500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:13.398006 12500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cac5ac36d7c414da3114351ed2f17ac, State: Initialized, Role: FOLLOWER
I20260810 03:25:13.398617 12500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac5ac36d7c414da3114351ed2f17ac" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33839 } }
I20260810 03:25:13.398890 12500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:13.399039 12500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:13.399221 12500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:13.401618 12500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac5ac36d7c414da3114351ed2f17ac" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33839 } }
I20260810 03:25:13.402221 12500 leader_election.cc:302] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cac5ac36d7c414da3114351ed2f17ac; no voters:
I20260810 03:25:13.402860 12500 leader_election.cc:288] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:13.403028 12511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:13.403920 12511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [term 1 LEADER]: Becoming Leader. State: Replica: 2cac5ac36d7c414da3114351ed2f17ac, State: Running, Role: LEADER
I20260810 03:25:13.404562 12511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac5ac36d7c414da3114351ed2f17ac" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33839 } }
I20260810 03:25:13.405509 12500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:13.407955 12512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cac5ac36d7c414da3114351ed2f17ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac5ac36d7c414da3114351ed2f17ac" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33839 } } }
I20260810 03:25:13.408402 12512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:13.410202 12513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cac5ac36d7c414da3114351ed2f17ac. Latest consensus state: current_term: 1 leader_uuid: "2cac5ac36d7c414da3114351ed2f17ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac5ac36d7c414da3114351ed2f17ac" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33839 } } }
I20260810 03:25:13.410573 12517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:13.410573 12513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:13.416675 12517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:13.428107 12517 catalog_manager.cc:1347] Generated new cluster ID: 4027bd29e1ea4818a2053f9c97a16a9d
I20260810 03:25:13.428292 12517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:13.439155 12517 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:13.440878 12517 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:13.453650 12517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cac5ac36d7c414da3114351ed2f17ac: Generated new TSK 0
I20260810 03:25:13.454695 12517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:25:13.465675 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12430
W20260810 03:25:13.466974 12495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36783
W20260810 03:25:13.467955 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12430: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:13.468145 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 12430
2026-08-10T03:25:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:13 dist-test-slave-btw8 krb5kdc[12536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:13 dist-test-slave-btw8 krb5kdc[12536](info): set up 2 sockets
Aug 10 03:25:13 dist-test-slave-btw8 krb5kdc[12536](info): commencing operation
krb5kdc: starting...
W20260810 03:25:14.685339 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:14 dist-test-slave-btw8 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:25:15.059865 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:38857
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:38857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:15.437377 12550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:15.437742 12550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:15.437902 12550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:15.445791 12550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:15.445967 12550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:15.446055 12550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:15.446149 12550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:15.446218 12550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:15.456750 12550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:38857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:15.458212 12550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:15.459672 12550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:15.473348 12556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:15.473369 12555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:15.473917 12558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:15.474442 12550 server_base.cc:1034] running on GCE node
I20260810 03:25:15.475481 12550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:15.477083 12550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:15.478344 12550 hybrid_clock.cc:648] HybridClock initialized: now 1786332315478306 us; error 36 us; skew 500 ppm
I20260810 03:25:15.478785 12550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:15 dist-test-slave-btw8 krb5kdc[12536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332315, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:15.482550 12550 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:25:15.484476 12550 webserver.cc:460] Webserver started at http://127.19.246.254:42093/ using document root <none> and password file <none>
I20260810 03:25:15.485273 12550 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:15.485394 12550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:15.485718 12550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:15.489312 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "8f40c2c2092648b3b620d07cf323381f"
format_stamp: "Formatted at 2026-08-10 03:25:15 on dist-test-slave-btw8"
server_key: "99dde826db1c09935f7226474a20f946"
server_key_iv: "f300c4863bbe162a55730d4eca8753f6"
server_key_version: "encryptionkey@0"
I20260810 03:25:15.490190 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "8f40c2c2092648b3b620d07cf323381f"
format_stamp: "Formatted at 2026-08-10 03:25:15 on dist-test-slave-btw8"
server_key: "99dde826db1c09935f7226474a20f946"
server_key_iv: "f300c4863bbe162a55730d4eca8753f6"
server_key_version: "encryptionkey@0"
I20260810 03:25:15.496229 12550 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 03:25:15.500475 12565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:15.502287 12550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 03:25:15.502503 12550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "8f40c2c2092648b3b620d07cf323381f"
format_stamp: "Formatted at 2026-08-10 03:25:15 on dist-test-slave-btw8"
server_key: "99dde826db1c09935f7226474a20f946"
server_key_iv: "f300c4863bbe162a55730d4eca8753f6"
server_key_version: "encryptionkey@0"
I20260810 03:25:15.502735 12550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:15.510048 12550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:15.515867 12550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:15.516185 12550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:15.537026 12550 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:38857
I20260810 03:25:15.537046 12620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:38857 every 8 connection(s)
I20260810 03:25:15.538609 12550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:15.539980 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 12550
I20260810 03:25:15.540174 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:15.540611 20443 external_mini_cluster.cc:1428] Setting key b3f7c20cf13623b975580c6d600ad36c
I20260810 03:25:15.543488 12621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:15.549037 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:15.542585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51049 (local address 127.19.246.254:38857)
0810 03:25:15.543523 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:15.543536 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:15.543584 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:25:15.545338 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:15.545362 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:15.545714 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:15.546140 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:15.547253 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:15.547259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:15.547271 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:15.547286 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:15.547428 (+ 142us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:15.547456 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:15.547697 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:15.547859 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 03:25:15.548048 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":353,"threads_started":1}
I20260810 03:25:15.553530 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:15.557783 12621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f: Bootstrap starting.
I20260810 03:25:15.561609 12621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:15.561849 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:15.559184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44763 (local address 127.19.246.254:38857)
0810 03:25:15.559418 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:15.559428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:15.559744 (+ 316us) server_negotiation.cc:407] Connection header received
0810 03:25:15.559956 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:15.559960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:15.560010 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:15.560119 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:15.561125 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:15.561129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:15.561132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:25:15.561148 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:15.561192 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:25:15.561207 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:15.561325 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:15.561421 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 03:25:15.561491 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:25:15.563216 12621 log.cc:824] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:15.567088 12621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f: No bootstrap required, opened a new log
I20260810 03:25:15.577373 12621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f40c2c2092648b3b620d07cf323381f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38857 } }
I20260810 03:25:15.577842 12621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:25:15.577927 12621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f40c2c2092648b3b620d07cf323381f, State: Initialized, Role: FOLLOWER
I20260810 03:25:15.578588 12621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f40c2c2092648b3b620d07cf323381f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38857 } }
I20260810 03:25:15.578833 12621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:25:15.578954 12621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:25:15.579098 12621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:15.582335 12621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f40c2c2092648b3b620d07cf323381f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38857 } }
I20260810 03:25:15.582942 12621 leader_election.cc:302] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f40c2c2092648b3b620d07cf323381f; no voters:
I20260810 03:25:15.583679 12621 leader_election.cc:288] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:25:15.583884 12633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:15.584645 12633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [term 1 LEADER]: Becoming Leader. State: Replica: 8f40c2c2092648b3b620d07cf323381f, State: Running, Role: LEADER
I20260810 03:25:15.585245 12633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f40c2c2092648b3b620d07cf323381f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38857 } }
I20260810 03:25:15.586113 12621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:15.588449 12635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f40c2c2092648b3b620d07cf323381f. Latest consensus state: current_term: 1 leader_uuid: "8f40c2c2092648b3b620d07cf323381f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f40c2c2092648b3b620d07cf323381f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38857 } } }
I20260810 03:25:15.588910 12635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:15.588613 12634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f40c2c2092648b3b620d07cf323381f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f40c2c2092648b3b620d07cf323381f" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 38857 } } }
I20260810 03:25:15.589213 12634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:15.595878 12639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:15.601944 12639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:15.612986 12639 catalog_manager.cc:1347] Generated new cluster ID: 61eb22d635784d3ea95ea9f9e79736ec
I20260810 03:25:15.613149 12639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:15.640628 12639 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:15.641155 12639 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:15.650440 12639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f40c2c2092648b3b620d07cf323381f: Generated new TSK 0
I20260810 03:25:15.651520 12639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:25:15.663455 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12550
W20260810 03:25:15.664635 12614 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51049
W20260810 03:25:15.665436 20443 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 03:25:15.665720 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 12550
2026-08-10T03:25:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2198 ms)
[----------] 18 tests from AuthTokenIssuingTest (41387 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:15 dist-test-slave-btw8 krb5kdc[12656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:15 dist-test-slave-btw8 krb5kdc[12656](info): set up 2 sockets
Aug 10 03:25:15 dist-test-slave-btw8 krb5kdc[12656](info): commencing operation
krb5kdc: starting...
W20260810 03:25:16.917008 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.003s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:17 dist-test-slave-btw8 krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:25:17.308436 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:33617
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:33617,127.19.246.253:41903,127.19.246.252:41099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:17.685027 12669 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:25:17.685493 12669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:17.685633 12669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:17.685777 12669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:17.693503 12669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:17.693683 12669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:17.693799 12669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:17.693893 12669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:17.693988 12669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:17.705132 12669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:33617,127.19.246.253:41903,127.19.246.252:41099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:33617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:17.706643 12669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:17.708173 12669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:17.721562 12677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:17.721652 12676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:17.721616 12679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:17.722786 12669 server_base.cc:1034] running on GCE node
I20260810 03:25:17.723742 12669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:17.725262 12669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:17.726516 12669 hybrid_clock.cc:648] HybridClock initialized: now 1786332317726431 us; error 54 us; skew 500 ppm
I20260810 03:25:17.726927 12669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:17 dist-test-slave-btw8 krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332317, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:17.730345 12669 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:25:17.732142 12669 webserver.cc:460] Webserver started at http://127.19.246.254:39721/ using document root <none> and password file <none>
I20260810 03:25:17.732911 12669 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:17.733057 12669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:17.733400 12669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:17.736672 12669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "1e9a24072a2843b0812e66ab18a9c944"
format_stamp: "Formatted at 2026-08-10 03:25:17 on dist-test-slave-btw8"
server_key: "018433d895f1cb3e1c40f9406ac4f04e"
server_key_iv: "75e44cb2a783f5638c3bf0221038426c"
server_key_version: "encryptionkey@0"
I20260810 03:25:17.737550 12669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "1e9a24072a2843b0812e66ab18a9c944"
format_stamp: "Formatted at 2026-08-10 03:25:17 on dist-test-slave-btw8"
server_key: "018433d895f1cb3e1c40f9406ac4f04e"
server_key_iv: "75e44cb2a783f5638c3bf0221038426c"
server_key_version: "encryptionkey@0"
I20260810 03:25:17.743374 12669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 03:25:17.747287 12687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:17.748971 12669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:25:17.749214 12669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "1e9a24072a2843b0812e66ab18a9c944"
format_stamp: "Formatted at 2026-08-10 03:25:17 on dist-test-slave-btw8"
server_key: "018433d895f1cb3e1c40f9406ac4f04e"
server_key_iv: "75e44cb2a783f5638c3bf0221038426c"
server_key_version: "encryptionkey@0"
I20260810 03:25:17.749482 12669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:17.779862 12669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:17.785689 12669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:17.786052 12669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:17.807715 12669 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:33617
I20260810 03:25:17.807881 12743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:33617 every 8 connection(s)
I20260810 03:25:17.809276 12669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:17.813392 12744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:17.818353 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 12669
I20260810 03:25:17.818602 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:17.819674 20443 external_mini_cluster.cc:1428] Setting key 2bae19f2bfdbe114366ad36a40eeda64
I20260810 03:25:17.823391 12744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:17 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332317, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:25:17.852869 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:17.829500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.19.246.254:33617)
0810 03:25:17.830332 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:17.830345 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:17.830416 (+ 71us) server_negotiation.cc:407] Connection header received
0810 03:25:17.831840 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:17.831852 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:17.832224 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:17.832473 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:17.833785 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:17.835990 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:17.838393 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:17.839057 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:17.839219 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:17.844112 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:17.844122 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:17.844136 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:17.844172 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:17.847732 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:17.848586 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:17.848597 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:17.848601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:17.848771 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:17.849220 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:17.849228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:17.849233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:17.849826 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:17.850082 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:17.850673 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 03:25:17.851081 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":380,"threads_started":1}
I20260810 03:25:17.852829 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:17.828047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:33617 (local address 127.0.0.1:53928)
0810 03:25:17.828981 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:17.829042 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:17.831335 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:17.832514 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:17.832519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:17.833164 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:17.833535 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:17.833552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:17.836213 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:17.836217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:17.838157 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:17.838191 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:17.839373 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:17.839384 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:17.839658 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:17.840647 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:17.840687 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:17.843832 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:17.847925 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:17.847941 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:17.847956 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:17.848429 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:17.848900 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:17.848906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:17.848911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:17.849095 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:17.849993 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:17.850002 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:17.850248 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 03:25:17.850539 (+ 291us) client_negotiation.cc:240] Negotiation successful
0810 03:25:17.851221 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":171,"thread_start_us":412,"threads_started":1}
WARNING: no policy specified for kudu/127.19.246.253@KRBTEST.COM; defaulting to no policy
I20260810 03:25:17.856843 12744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:25:17.858139 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:17.857661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44398)
0810 03:25:17.857871 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:17.857999 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 03:25:17.858453 12689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.253@KRBTEST.COM" created.
I20260810 03:25:17.860284 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:17.859878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44410)
0810 03:25:17.860036 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:17.860121 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 03:25:17.867758 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:41903: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab.
Entry for principal kudu/127.19.246.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab.
I20260810 03:25:17.904109 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.253:41903
--webserver_interface=127.19.246.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab
--principal=kudu/127.19.246.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:33617,127.19.246.253:41903,127.19.246.252:41099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:25:17.917320 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } attempt: 1
I20260810 03:25:17.918871 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:17.918198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44412)
0810 03:25:17.918475 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:17.918676 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 03:25:17.920290 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:17.919686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44414)
0810 03:25:17.919947 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:17.920056 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260810 03:25:17.920667 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:41903: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
I20260810 03:25:17.988981 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } attempt: 2
I20260810 03:25:17.990262 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:17.989736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44430)
0810 03:25:17.990018 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:17.990108 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 03:25:17.991335 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:17.990952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44444)
0810 03:25:17.991118 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:17.991187 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 03:25:17.991585 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:41903: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
I20260810 03:25:18.082899 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } attempt: 3
I20260810 03:25:18.084354 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.083859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44454)
0810 03:25:18.084080 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.084193 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 03:25:18.085793 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.085324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44464)
0810 03:25:18.085497 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.085584 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 03:25:18.086098 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:41903: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
I20260810 03:25:18.229429 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } attempt: 4
I20260810 03:25:18.231091 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.230355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44476)
0810 03:25:18.230774 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.230902 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260810 03:25:18.232404 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.232012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44478)
0810 03:25:18.232167 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.232267 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 03:25:18.232739 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:41903: Network error: Client connection negotiation failed: client connection to 127.19.246.253:41903: connect: Connection refused (error 111)
W20260810 03:25:18.268260 12752 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:25:18.268690 12752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:18.268818 12752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:18.268954 12752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:18.276788 12752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:18.276906 12752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:18.277006 12752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:18.277112 12752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:18.277220 12752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:18.288126 12752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:33617,127.19.246.253:41903,127.19.246.252:41099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab
--principal=kudu/127.19.246.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.253:41903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.246.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:18.289662 12752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:18.291060 12752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:18.303495 12761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:18.303439 12759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:18.303478 12758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:18.304956 12752 server_base.cc:1034] running on GCE node
I20260810 03:25:18.305923 12752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:18.307693 12752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:18.309021 12752 hybrid_clock.cc:648] HybridClock initialized: now 1786332318308918 us; error 71 us; skew 500 ppm
I20260810 03:25:18.309480 12752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:18 dist-test-slave-btw8 krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:18.318423 12752 init.cc:377] Logged in from keytab as kudu/127.19.246.253@KRBTEST.COM (short username kudu)
I20260810 03:25:18.320642 12752 webserver.cc:460] Webserver started at http://127.19.246.253:42469/ using document root <none> and password file <none>
I20260810 03:25:18.321503 12752 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:18.321669 12752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:18.322024 12752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:18.325450 12752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/data/instance:
uuid: "f0823abb74d546f098359d609c53e58e"
format_stamp: "Formatted at 2026-08-10 03:25:18 on dist-test-slave-btw8"
server_key: "8913b43b6c2cdceac7ce8bec973010fa"
server_key_iv: "c14ca18be483ba15c7521a9ca5b1658c"
server_key_version: "encryptionkey@0"
I20260810 03:25:18.326356 12752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/wal/instance:
uuid: "f0823abb74d546f098359d609c53e58e"
format_stamp: "Formatted at 2026-08-10 03:25:18 on dist-test-slave-btw8"
server_key: "8913b43b6c2cdceac7ce8bec973010fa"
server_key_iv: "c14ca18be483ba15c7521a9ca5b1658c"
server_key_version: "encryptionkey@0"
I20260810 03:25:18.332458 12752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 03:25:18.336699 12769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:18.338375 12752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:25:18.338641 12752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/wal
uuid: "f0823abb74d546f098359d609c53e58e"
format_stamp: "Formatted at 2026-08-10 03:25:18 on dist-test-slave-btw8"
server_key: "8913b43b6c2cdceac7ce8bec973010fa"
server_key_iv: "c14ca18be483ba15c7521a9ca5b1658c"
server_key_version: "encryptionkey@0"
I20260810 03:25:18.338922 12752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:18.359863 12752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:18.364723 12752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:18.365036 12752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:18.393347 12752 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.253:41903
I20260810 03:25:18.393368 12823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.253:41903 every 8 connection(s)
I20260810 03:25:18.394999 12752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
I20260810 03:25:18.398034 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 12752
I20260810 03:25:18.398301 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-1/wal/instance
I20260810 03:25:18.398866 20443 external_mini_cluster.cc:1428] Setting key a3399e114606f6c0ede4a1c6bd1a3ad0
I20260810 03:25:18.399302 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:18.406469 12824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:18 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.246.252@KRBTEST.COM; defaulting to no policy
I20260810 03:25:18.438999 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.409699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.19.246.254:33617)
0810 03:25:18.415219 (+ 5520us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:18.415229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:18.416004 (+ 775us) server_negotiation.cc:407] Connection header received
0810 03:25:18.419381 (+ 3377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:18.419393 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:18.419519 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:18.419628 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:18.422230 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.423055 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.425823 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.426290 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.426404 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:18.431807 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:18.431814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:18.431817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:18.431867 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:18.435110 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:18.436218 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:18.436222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:18.436225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:18.436374 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:18.436880 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:18.436885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:18.436887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:18.437211 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:18.437384 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:18.438284 (+ 900us) server_negotiation.cc:299] Negotiation successful
0810 03:25:18.438587 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5344,"thread_start_us":5182,"threads_started":1}
I20260810 03:25:18.440208 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.410405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:33617 (local address 127.0.0.1:53938)
0810 03:25:18.415766 (+ 5361us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.415827 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:18.418910 (+ 3083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:18.419997 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:18.420013 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:18.420906 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:18.421904 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:18.421935 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.423263 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.423271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:18.425607 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:18.425639 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.426596 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.426611 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:18.427019 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:18.427989 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:18.428027 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:18.431482 (+ 3455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:18.435336 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:18.435347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:18.435406 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:18.436026 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:18.436506 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:18.436512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:18.436515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:18.436737 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:18.437354 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:18.437363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:18.437868 (+ 505us) client_negotiation.cc:769] Sending connection context
0810 03:25:18.438156 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 03:25:18.438805 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5008,"thread_start_us":4697,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.252@KRBTEST.COM" created.
I20260810 03:25:18.442348 12824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:18 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:25:18.461210 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.443297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44486)
0810 03:25:18.443548 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.443573 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:18.443704 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:18.444547 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:18.444555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:18.445112 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:18.445334 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:18.445350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.446859 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.446867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:18.448384 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:18.448404 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.449398 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.449406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:18.449648 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:18.450460 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:18.450490 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:18.453810 (+ 3320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:18.458520 (+ 4710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:18.458525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:18.458528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:18.458960 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:18.459492 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:18.459496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:18.459499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:18.459640 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:18.460449 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:18.460453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:18.460570 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 03:25:18.460683 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 03:25:18.460919 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 03:25:18.461766 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.443332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.19.246.253:41903)
0810 03:25:18.444013 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:18.444019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:18.444054 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:25:18.444105 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:18.444112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:18.444371 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:18.444502 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:18.445552 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.446697 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.448591 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.449225 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.449338 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:18.454113 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:18.454133 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:18.454151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:18.454195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:18.458284 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:18.459145 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:18.459158 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:18.459164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:18.459331 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:18.459771 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:18.459779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:18.459784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:18.460311 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:18.460537 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:18.460898 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 03:25:18.461234 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":320,"threads_started":1}
I20260810 03:25:18.464464 12824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:25:18.465899 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.465398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41636)
0810 03:25:18.465604 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.465735 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 03:25:18.466212 12773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.468001 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.467596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41644)
0810 03:25:18.467798 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.467884 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 03:25:18.468343 12824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab.
Entry for principal kudu/127.19.246.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab.
I20260810 03:25:18.488448 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.252:41099
--webserver_interface=127.19.246.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab
--principal=kudu/127.19.246.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:33617,127.19.246.253:41903,127.19.246.252:41099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:25:18.517732 12824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 1
I20260810 03:25:18.519050 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.518470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41658)
0810 03:25:18.518680 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.518865 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 03:25:18.520148 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.519753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41666)
0810 03:25:18.519940 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.520021 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 03:25:18.520357 12824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.532043 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } attempt: 5
Aug 10 03:25:18 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332317, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:25:18.548689 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.533033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.19.246.253:41903)
0810 03:25:18.533246 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:18.533254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:18.533277 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:25:18.533463 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:18.533470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:18.533579 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:18.533691 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:18.534704 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.535824 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.537113 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.537788 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.537925 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:18.541985 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:18.541993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:18.541998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:18.542044 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:18.545792 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:18.546598 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:18.546604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:18.546608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:18.546769 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:18.547148 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:18.547153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:18.547156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:18.547551 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:18.547671 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:18.548235 (+ 564us) server_negotiation.cc:299] Negotiation successful
0810 03:25:18.548444 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:25:18.548909 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.532894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44494)
0810 03:25:18.533164 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.533192 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:18.533308 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:18.533801 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:18.533806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:18.534267 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:18.534502 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:18.534513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.535987 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.535992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:18.536929 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:18.536945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.537998 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:18.538004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:18.538222 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:18.539033 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:18.539059 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:18.541722 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:18.546012 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:18.546017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:18.546020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:18.546459 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:18.546891 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:18.546894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:18.546896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:18.547030 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:18.547702 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:18.547707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:18.547914 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 03:25:18.548132 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 03:25:18.548466 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 03:25:18.550247 12744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:25:18.551538 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.550986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41682)
0810 03:25:18.551187 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.551323 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 03:25:18.552764 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.552334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41694)
0810 03:25:18.552502 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.552592 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 03:25:18.553022 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.588692 12824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 2
I20260810 03:25:18.590227 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.589688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41708)
0810 03:25:18.589942 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.590052 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 03:25:18.591622 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.591028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41714)
0810 03:25:18.591315 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.591472 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 03:25:18.591871 12824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.604311 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 1
I20260810 03:25:18.605782 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.605149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41726)
0810 03:25:18.605417 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.605551 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 03:25:18.607043 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.606687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41742)
0810 03:25:18.606846 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.606925 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 03:25:18.607424 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.675752 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 2
I20260810 03:25:18.677183 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.676656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41756)
0810 03:25:18.676905 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.677002 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 03:25:18.678313 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.677966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41766)
0810 03:25:18.678115 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.678191 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 03:25:18.678593 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.683110 12824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 3
I20260810 03:25:18.684435 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.683865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41782)
0810 03:25:18.684090 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.684254 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 03:25:18.685663 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.685190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41792)
0810 03:25:18.685435 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.685530 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260810 03:25:18.685899 12824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.784884 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 3
I20260810 03:25:18.786230 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.785695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41806)
0810 03:25:18.785926 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.786025 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 03:25:18.787591 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.787103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41822)
0810 03:25:18.787326 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.787472 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260810 03:25:18.787909 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.829152 12824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 4
I20260810 03:25:18.830379 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.829877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41824)
0810 03:25:18.830127 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.830217 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 03:25:18.831594 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.831168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41838)
0810 03:25:18.831404 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.831486 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260810 03:25:18.831892 12824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
W20260810 03:25:18.870535 12833 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:25:18.870909 12833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:18.870975 12833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:18.871060 12833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:18.878799 12833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:18.878924 12833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:18.878986 12833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:18.879031 12833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:18.879092 12833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:18.889883 12833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:33617,127.19.246.253:41903,127.19.246.252:41099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab
--principal=kudu/127.19.246.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.252:41099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.246.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:18.891278 12833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:18.892668 12833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:18.906200 12839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:18.906241 12838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:18.907020 12841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:18.907063 12833 server_base.cc:1034] running on GCE node
I20260810 03:25:18.908063 12833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:18.909744 12833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:18.911098 12833 hybrid_clock.cc:648] HybridClock initialized: now 1786332318911024 us; error 38 us; skew 500 ppm
I20260810 03:25:18.911574 12833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:18 dist-test-slave-btw8 krb5kdc[12656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:18.915150 12833 init.cc:377] Logged in from keytab as kudu/127.19.246.252@KRBTEST.COM (short username kudu)
I20260810 03:25:18.916987 12833 webserver.cc:460] Webserver started at http://127.19.246.252:37451/ using document root <none> and password file <none>
I20260810 03:25:18.917934 12833 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:18.918087 12833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:18.918437 12833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:18.921875 12833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/data/instance:
uuid: "e1a7d73353cd4769ac21f159619d90a6"
format_stamp: "Formatted at 2026-08-10 03:25:18 on dist-test-slave-btw8"
server_key: "9a73787ab9daa191098561db25434ade"
server_key_iv: "c7a9e2f17c4315dde3941d0cc1bfea9b"
server_key_version: "encryptionkey@0"
I20260810 03:25:18.922821 12833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/wal/instance:
uuid: "e1a7d73353cd4769ac21f159619d90a6"
format_stamp: "Formatted at 2026-08-10 03:25:18 on dist-test-slave-btw8"
server_key: "9a73787ab9daa191098561db25434ade"
server_key_iv: "c7a9e2f17c4315dde3941d0cc1bfea9b"
server_key_version: "encryptionkey@0"
I20260810 03:25:18.929391 12833 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 03:25:18.933522 12848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:18.935190 12833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:25:18.935453 12833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/wal
uuid: "e1a7d73353cd4769ac21f159619d90a6"
format_stamp: "Formatted at 2026-08-10 03:25:18 on dist-test-slave-btw8"
server_key: "9a73787ab9daa191098561db25434ade"
server_key_iv: "c7a9e2f17c4315dde3941d0cc1bfea9b"
server_key_version: "encryptionkey@0"
I20260810 03:25:18.935708 12833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:18.956735 12833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:18.961120 12833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:18.961424 12833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:18.965160 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 4
I20260810 03:25:18.966348 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.965892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41842)
0810 03:25:18.966093 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.966187 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 03:25:18.967708 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.967164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41846)
0810 03:25:18.967399 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:18.967512 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260810 03:25:18.967952 12744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:41099: Network error: Client connection negotiation failed: client connection to 127.19.246.252:41099: connect: Connection refused (error 111)
I20260810 03:25:18.982520 12833 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.252:41099
I20260810 03:25:18.982560 12903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.252:41099 every 8 connection(s)
I20260810 03:25:18.984105 12833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
I20260810 03:25:18.987946 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 12833
I20260810 03:25:18.988161 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786331905428759-20443-0/minicluster-data/master-2/wal/instance
I20260810 03:25:18.988636 20443 external_mini_cluster.cc:1428] Setting key b059525093f08bbb23af4bf10f6960f4
I20260810 03:25:18.988636 12904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:18.999833 12904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:18 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
Aug 10 03:25:19 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:25:19.015784 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:18.990571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.19.246.252:41099)
0810 03:25:18.991472 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:18.991484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:18.991526 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:25:18.993858 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:18.993882 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:18.994317 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:18.994679 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:18.996875 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:18.998277 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.000471 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.001369 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.001529 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.007268 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.007276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.007288 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.007334 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.011110 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.012047 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.012054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.012061 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.012243 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.012667 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.012681 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.012683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.013322 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.013684 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.014152 (+ 468us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.014605 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":390,"threads_started":1}
I20260810 03:25:19.021402 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.003325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:33617 (local address 127.0.0.1:53948)
0810 03:25:19.004516 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:19.004570 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:19.004773 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:19.005215 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:19.005227 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:19.006064 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:19.006474 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.006493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.007864 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.007877 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.009463 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.009484 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.010408 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.010415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.010639 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.011392 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:19.011426 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:19.013882 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:19.017979 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.017988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.017993 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.018619 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.019109 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.019116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.019122 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.019306 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.020329 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:19.020335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:19.020496 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 03:25:19.020747 (+ 251us) client_negotiation.cc:240] Negotiation successful
0810 03:25:19.021090 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":610,"threads_started":1}
I20260810 03:25:19.022455 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.003411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.19.246.254:33617)
0810 03:25:19.004061 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.004067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.004716 (+ 649us) server_negotiation.cc:407] Connection header received
0810 03:25:19.004905 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.004909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.004984 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.005074 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.006692 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.007674 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.009690 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.010145 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.010355 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.014134 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.014140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.014144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.014194 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.017815 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.018775 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.018782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.018788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.018974 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.019534 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.019539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.019544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.019884 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.020253 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.021773 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.021977 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":303,"threads_started":1}
I20260810 03:25:19.023983 12904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:19 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
Aug 10 03:25:19 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:25:19.038609 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.019607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.19.246.254:33617)
0810 03:25:19.020447 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.020452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.020471 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:25:19.020511 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.020514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.020565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.020694 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.022303 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.023430 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.024937 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.025591 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.025687 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.031679 (+ 5992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.031685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.031689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.031728 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.035135 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.036148 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.036153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.036155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.036304 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.036777 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.036785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.036791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.037226 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.037519 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.037864 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.038103 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":477,"threads_started":1}
I20260810 03:25:19.039747 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.024903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.19.246.253:41903)
0810 03:25:19.025114 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.025120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.025139 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:25:19.025201 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.025209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.025288 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.025392 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.026248 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.027096 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.029200 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.029635 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.029873 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.033224 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.033230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.033233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.033269 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.036333 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.037235 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.037239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.037241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.037417 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.037929 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.037935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.037940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.038293 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.038414 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.039009 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.039309 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:25:19.040177 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.024770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:41903 (local address 127.0.0.1:44502)
0810 03:25:19.024967 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:19.024988 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:19.025087 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:19.025423 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:19.025427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:19.025874 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:19.026077 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.026089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.027272 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.027280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.029012 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.029031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.029888 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.029894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.030086 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.030787 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:19.030805 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:19.033011 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:19.036481 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.036486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.036488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.037096 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.037571 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.037577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.037580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.037794 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.038411 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:19.038417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:19.038619 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 03:25:19.039458 (+ 839us) client_negotiation.cc:240] Negotiation successful
0810 03:25:19.039776 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 03:25:19.041499 12904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:19 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
Aug 10 03:25:19 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:25:19.056002 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.042479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41870)
0810 03:25:19.042814 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:19.042834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:19.042943 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:19.043303 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:19.043307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:19.043790 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:19.043931 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.043940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.045005 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.045013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.045863 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.045875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.046573 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.046579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.046798 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.047323 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:19.047340 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:19.049793 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:19.053394 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.053401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.053405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.053797 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.054181 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.054185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.054189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.054369 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.054897 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:19.054902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:19.055059 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 03:25:19.055206 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 03:25:19.055568 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260810 03:25:19.056066 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.042716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.19.246.252:41099)
0810 03:25:19.042866 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.042874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.042893 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:25:19.043078 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.043083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.043151 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.043287 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.044073 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.044881 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.046002 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.046411 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.046556 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.050007 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.050011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.050014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.050043 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.053261 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.053918 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.053925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.053928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.054055 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.054471 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.054478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.054481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.054771 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.054920 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.055306 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.055739 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":53}
I20260810 03:25:19.057139 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.040991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.19.246.253:41903)
0810 03:25:19.041165 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.041174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.041197 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:25:19.041337 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.041342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.041439 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.041541 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.043209 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.044289 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.045642 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.046118 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.046270 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.051315 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.051321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.051324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.051412 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.054135 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.054950 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.054955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.054959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.055136 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.055599 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.055605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.055608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.055895 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.056199 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.056534 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.056803 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 03:25:19.062373 12904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: Bootstrap starting.
I20260810 03:25:19.062733 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:19.066483 12904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:19.067994 12904 log.cc:824] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:19.066171 12719 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1e9a24072a2843b0812e66ab18a9c944"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53964
I20260810 03:25:19.072140 12904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: No bootstrap required, opened a new log
I20260810 03:25:19.085217 12904 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } }
I20260810 03:25:19.085777 12904 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a7d73353cd4769ac21f159619d90a6, State: Initialized, Role: FOLLOWER
I20260810 03:25:19.086647 12904 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } }
I20260810 03:25:19.088037 12912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.088444 12912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:19.089744 12904 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:25:19.100639 12924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:25:19.100816 12924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:25:19.098094 12833 master_runner.cc:186] Error getting master registration for 127.19.246.254:33617: OK, instance_id { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" instance_seqno: 1786332317801667 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:25:19.106617 12833 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:25:19.131191 12824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 5
Aug 10 03:25:19 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332318, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
I20260810 03:25:19.149396 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.132119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41886)
0810 03:25:19.132332 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:19.132356 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:19.132504 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:19.133007 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:19.133014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:19.133600 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:19.133825 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.133836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.135384 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.135389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.136806 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.136824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.137949 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.137954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.138201 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.138951 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:19.138974 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:19.142054 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:19.146462 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.146471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.146474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.146874 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.147379 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.147384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.147388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.147596 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.148322 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:19.148326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:19.148495 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 03:25:19.148644 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 03:25:19.148985 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:25:19.149418 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.132375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.19.246.252:41099)
0810 03:25:19.132619 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.132625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.132648 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:25:19.132726 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.132732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.132816 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.132927 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.134008 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.135190 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.136977 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.137774 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.137914 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.142324 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.142332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.142341 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.142386 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.146304 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.147013 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.147020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.147024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.147181 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.147730 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.147735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.147739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.148187 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.148372 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.148800 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.149041 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:25:19.156104 12824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: Bootstrap starting.
I20260810 03:25:19.160555 12824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:19.161980 12824 log.cc:824] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:19.165740 12824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: No bootstrap required, opened a new log
I20260810 03:25:19.177402 12824 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } }
I20260810 03:25:19.177910 12824 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0823abb74d546f098359d609c53e58e, State: Initialized, Role: FOLLOWER
I20260810 03:25:19.178491 12824 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } }
I20260810 03:25:19.179724 12928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.180195 12928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:19.181083 12824 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:25:19.198611 12941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:25:19.198936 12941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:25:19.189364 12752 master_runner.cc:186] Error getting master registration for 127.19.246.254:33617: OK, instance_id { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" instance_seqno: 1786332317801667 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:25:19.205857 12752 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 03:25:19.213743 12856 tablet.cc:2369] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:19.245224 12744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } attempt: 5
Aug 10 03:25:19 dist-test-slave-btw8 krb5kdc[12656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332317, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
I20260810 03:25:19.265825 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.245980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:41099 (local address 127.0.0.1:41902)
0810 03:25:19.246323 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:19.246346 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:19.246493 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:19.247411 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:19.247418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:19.248042 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:19.248280 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.248296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.249893 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.249900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.251508 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:19.251528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.252883 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.252890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:19.253106 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.254009 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:19.254047 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:19.257259 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:19.262466 (+ 5207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.262473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.262478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.263129 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.263711 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:19.263717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:19.263721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:19.263924 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:19.264613 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:19.264619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:19.264846 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 03:25:19.265011 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 03:25:19.265376 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260810 03:25:19.265853 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.246117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.19.246.252:41099)
0810 03:25:19.246765 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.246771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.246791 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:25:19.246857 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.246862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.246943 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.247065 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.248496 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.249694 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.251719 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.252465 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.252613 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.257571 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.257580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.257586 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.257638 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.262234 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.263312 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.263320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.263323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.263552 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.264053 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.264072 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.264076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.264476 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.264646 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.265184 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.265461 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508}
I20260810 03:25:19.273458 12744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944: Bootstrap starting.
I20260810 03:25:19.278580 12744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:19.281116 12744 log.cc:824] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:19.288959 12744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944: No bootstrap required, opened a new log
I20260810 03:25:19.304005 12744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } }
I20260810 03:25:19.305025 12744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9a24072a2843b0812e66ab18a9c944, State: Initialized, Role: FOLLOWER
I20260810 03:25:19.306116 12744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } }
I20260810 03:25:19.307789 12944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.308288 12944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:19.310631 12719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 03:25:19.310854 12719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:19.311785 12744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:19.314694 12719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } }
I20260810 03:25:19.317749 12797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e9a24072a2843b0812e66ab18a9c944" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f0823abb74d546f098359d609c53e58e"
I20260810 03:25:19.318346 12797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:19.322563 12797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e9a24072a2843b0812e66ab18a9c944 in term 1.
I20260810 03:25:19.323616 12689 leader_election.cc:302] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e9a24072a2843b0812e66ab18a9c944, f0823abb74d546f098359d609c53e58e; no voters:
I20260810 03:25:19.324255 12944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:19.325675 12878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e9a24072a2843b0812e66ab18a9c944" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e1a7d73353cd4769ac21f159619d90a6"
I20260810 03:25:19.326283 12878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:19.330498 12878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e9a24072a2843b0812e66ab18a9c944 in term 1.
I20260810 03:25:19.331700 12944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [term 1 LEADER]: Becoming Leader. State: Replica: 1e9a24072a2843b0812e66ab18a9c944, State: Running, Role: LEADER
I20260810 03:25:19.331799 12719 leader_election.cc:288] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [CANDIDATE]: Term 1 election: Requested vote from peers f0823abb74d546f098359d609c53e58e (127.19.246.253:41903), e1a7d73353cd4769ac21f159619d90a6 (127.19.246.252:41099)
I20260810 03:25:19.332468 12944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } }
I20260810 03:25:19.347887 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e9a24072a2843b0812e66ab18a9c944. Latest consensus state: current_term: 1 leader_uuid: "1e9a24072a2843b0812e66ab18a9c944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.348616 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:19.364010 12950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260810 03:25:19.368067 12775 tablet.cc:2369] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:19.385528 12950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:25:19.437723 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.381441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.19.246.254:33617)
0810 03:25:19.382809 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.382816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.382842 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:25:19.382919 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.382924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.383040 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.383161 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.388507 (+ 5346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.390192 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.404825 (+ 14633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.405626 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.405763 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.415568 (+ 9805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.415576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.415580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.415631 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.430005 (+ 14374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.430968 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.430974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.430979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.431253 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.431960 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.431966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.431970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.432437 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.436720 (+ 4283us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.436920 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.437245 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211}
I20260810 03:25:19.448558 12878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [term 1 FOLLOWER]: Refusing update from remote peer 1e9a24072a2843b0812e66ab18a9c944: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:19.450146 12946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:19.454092 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.382705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.19.246.253:41903)
0810 03:25:19.382893 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.382899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.411331 (+ 28432us) server_negotiation.cc:407] Connection header received
0810 03:25:19.412727 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.412735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.412838 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.412928 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.418090 (+ 5162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.420596 (+ 2506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.427560 (+ 6964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.428345 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.428484 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.436487 (+ 8003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.436497 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.436503 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.436565 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.440659 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.444097 (+ 3438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.444103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.444106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.444329 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.444873 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.444881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.444884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.448864 (+ 3980us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.449158 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.453196 (+ 4038us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.453542 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:25:19.458982 12797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [term 1 FOLLOWER]: Refusing update from remote peer 1e9a24072a2843b0812e66ab18a9c944: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:19.460246 12946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:19.462296 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:19.385759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.19.246.252:41099)
0810 03:25:19.385997 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.386004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.412352 (+ 26348us) server_negotiation.cc:407] Connection header received
0810 03:25:19.415443 (+ 3091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.415453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.415594 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.415679 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.418599 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.419793 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.427653 (+ 7860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:19.428350 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:19.428470 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:19.444499 (+ 16029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:19.444506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:19.444511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:19.444561 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:19.448453 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.458348 (+ 9895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.458354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.458357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.458574 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:19.459705 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:19.459712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:19.459716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:19.460195 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:19.460460 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:19.461615 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0810 03:25:19.461918 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 03:25:19.489800 12912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e9a24072a2843b0812e66ab18a9c944. Latest consensus state: current_term: 1 leader_uuid: "1e9a24072a2843b0812e66ab18a9c944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.490170 12912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:19.491323 12928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e9a24072a2843b0812e66ab18a9c944. Latest consensus state: current_term: 1 leader_uuid: "1e9a24072a2843b0812e66ab18a9c944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.491703 12928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:19.492787 12959 mvcc.cc:205] Tried to move back new op lower bound from 7316817180438294528 to 7316817179991764992. Current Snapshot: MvccSnapshot[applied={T|T < 7316817180438294528}]
I20260810 03:25:19.504873 12912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e9a24072a2843b0812e66ab18a9c944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.505208 12912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:19.509061 12928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e9a24072a2843b0812e66ab18a9c944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.509361 12928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:19.510537 12950 catalog_manager.cc:1347] Generated new cluster ID: 63478b33ee134fcab4ef548abaa120f8
I20260810 03:25:19.510659 12950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:19.512454 12964 mvcc.cc:205] Tried to move back new op lower bound from 7316817180438294528 to 7316817179991764992. Current Snapshot: MvccSnapshot[applied={T|T < 7316817180438294528 or (T in {7316817180438294528})}]
I20260810 03:25:19.527530 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e9a24072a2843b0812e66ab18a9c944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.527982 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:19.527678 12944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e9a24072a2843b0812e66ab18a9c944" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e9a24072a2843b0812e66ab18a9c944" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 33617 } } peers { permanent_uuid: "f0823abb74d546f098359d609c53e58e" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 41903 } } peers { permanent_uuid: "e1a7d73353cd4769ac21f159619d90a6" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 41099 } } }
I20260810 03:25:19.530951 12944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:19.555442 12950 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:19.557819 12950 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:19.575433 12976 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:25:19.583666 12950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e9a24072a2843b0812e66ab18a9c944: Generated new TSK 0
I20260810 03:25:19.590859 12950 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:25:19.752550 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.736365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.19.246.253:41903)
0810 03:25:19.736591 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.736598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.750039 (+ 13441us) server_negotiation.cc:407] Connection header received
0810 03:25:19.750635 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.750642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.750787 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.750879 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.752280 (+ 1401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44534: BlockingRecv error: recv got EOF from 127.0.0.1:44534 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260810 03:25:19.753749 12910 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.734046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.19.246.254:33617)
0810 03:25:19.734322 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.734331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.741767 (+ 7436us) server_negotiation.cc:407] Connection header received
0810 03:25:19.751797 (+ 10030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.751806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.751909 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.752015 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.753374 (+ 1359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53990: BlockingRecv error: recv got EOF from 127.0.0.1:53990 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 03:25:19.755901 12907 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.746394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.19.246.252:41099)
0810 03:25:19.746609 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.746615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.752006 (+ 5391us) server_negotiation.cc:407] Connection header received
0810 03:25:19.753992 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.753997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.754085 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.754170 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.755613 (+ 1443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41922: BlockingRecv error: recv got EOF from 127.0.0.1:41922 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260810 03:25:19.798486 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.788328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.19.246.253:41903)
0810 03:25:19.788568 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.788576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.796125 (+ 7549us) server_negotiation.cc:407] Connection header received
0810 03:25:19.796547 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.796555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.796723 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.796825 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.798150 (+ 1325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44542: BlockingRecv error: recv got EOF from 127.0.0.1:44542 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260810 03:25:19.855505 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.841827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.19.246.253:41903)
0810 03:25:19.842090 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.842104 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.853228 (+ 11124us) server_negotiation.cc:407] Connection header received
0810 03:25:19.853670 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.853678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.853794 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.853881 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.855177 (+ 1296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44558: BlockingRecv error: recv got EOF from 127.0.0.1:44558 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260810 03:25:19.912243 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.892423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.19.246.253:41903)
0810 03:25:19.892641 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.892653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.909984 (+ 17331us) server_negotiation.cc:407] Connection header received
0810 03:25:19.910375 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.910387 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.910523 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.910626 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.911935 (+ 1309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44566: BlockingRecv error: recv got EOF from 127.0.0.1:44566 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260810 03:25:19.923122 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.920587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.19.246.253:41903)
0810 03:25:19.920778 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.920784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.921408 (+ 624us) server_negotiation.cc:407] Connection header received
0810 03:25:19.921605 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.921612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.921727 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.921932 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.922858 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44582: BlockingRecv error: recv got EOF from 127.0.0.1:44582 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 03:25:19.940351 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.936248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.19.246.253:41903)
0810 03:25:19.937899 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.937906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.938491 (+ 585us) server_negotiation.cc:407] Connection header received
0810 03:25:19.938758 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.938765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.938877 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.939006 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.940042 (+ 1036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44598: BlockingRecv error: recv got EOF from 127.0.0.1:44598 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260810 03:25:19.968281 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:19.965239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.19.246.253:41903)
0810 03:25:19.965556 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:19.965563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:19.966189 (+ 626us) server_negotiation.cc:407] Connection header received
0810 03:25:19.966517 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:19.966524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:19.966625 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:19.966706 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:19.967970 (+ 1264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44606: BlockingRecv error: recv got EOF from 127.0.0.1:44606 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 03:25:20.006831 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:20.004334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.19.246.253:41903)
0810 03:25:20.004572 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:20.004581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:20.005186 (+ 605us) server_negotiation.cc:407] Connection header received
0810 03:25:20.005421 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:20.005426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:20.005542 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:20.005652 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:20.006534 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44618: BlockingRecv error: recv got EOF from 127.0.0.1:44618 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 03:25:20.077656 12830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:20.075203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.19.246.253:41903)
0810 03:25:20.075456 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:20.075463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:20.075987 (+ 524us) server_negotiation.cc:407] Connection header received
0810 03:25:20.076376 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:20.076384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:20.076503 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:20.076625 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:20.077406 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44630: BlockingRecv error: recv got EOF from 127.0.0.1:44630 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:25:20.103065 12924 catalog_manager.cc:1259] Loaded cluster ID: 63478b33ee134fcab4ef548abaa120f8
I20260810 03:25:20.103178 12924 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: loading cluster ID for follower catalog manager: success
I20260810 03:25:20.106038 12924 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: acquiring CA information for follower catalog manager: success
I20260810 03:25:20.108223 12924 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1a7d73353cd4769ac21f159619d90a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:25:20.201465 12941 catalog_manager.cc:1259] Loaded cluster ID: 63478b33ee134fcab4ef548abaa120f8
I20260810 03:25:20.201637 12941 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: loading cluster ID for follower catalog manager: success
I20260810 03:25:20.204793 12941 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: acquiring CA information for follower catalog manager: success
I20260810 03:25:20.206990 12941 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0823abb74d546f098359d609c53e58e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:25:20.217278 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:20.209808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.19.246.253:41903)
0810 03:25:20.209994 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:20.209999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:20.210449 (+ 450us) server_negotiation.cc:407] Connection header received
0810 03:25:20.210767 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:20.210772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:20.210940 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:20.211065 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:20.211899 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.212851 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.214289 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.214827 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.214934 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:20.215782 (+ 848us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:20.216379 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:20.216503 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:20.216827 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 03:25:20.216907 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:25:20.230541 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:20.224330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.19.246.252:41099)
0810 03:25:20.224593 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:20.224602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:20.224966 (+ 364us) server_negotiation.cc:407] Connection header received
0810 03:25:20.225252 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:20.225257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:20.225434 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:20.225612 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:20.226030 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.226773 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.228250 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.228642 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.228799 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:20.229317 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:20.229890 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:20.230059 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:20.230122 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 03:25:20.230220 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 03:25:20.246907 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:20.238652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.19.246.254:33617)
0810 03:25:20.238942 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:20.238952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:20.239674 (+ 722us) server_negotiation.cc:407] Connection header received
0810 03:25:20.240069 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:20.240073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:20.240312 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:20.240547 (+ 235us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:20.241072 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.241946 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.243863 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.244538 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.244661 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:20.245628 (+ 967us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:20.246125 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:20.246250 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:20.246443 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 03:25:20.246546 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 03:25:20.250939 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:20.239439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.19.246.252:41099)
0810 03:25:20.239793 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:20.239799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:20.243844 (+ 4045us) server_negotiation.cc:407] Connection header received
0810 03:25:20.244274 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:20.244304 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:20.244415 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:20.244535 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:20.245185 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.246203 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.248479 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.249123 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.249264 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:20.249948 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:20.250107 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:20.250199 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:20.250459 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 03:25:20.250588 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260810 03:25:20.252077 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:20.238754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.19.246.253:41903)
0810 03:25:20.239019 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:20.239028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:20.243982 (+ 4954us) server_negotiation.cc:407] Connection header received
0810 03:25:20.244372 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:20.244379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:20.244499 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:20.244598 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:20.245655 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.246641 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.248816 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:20.249523 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:20.249689 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:20.250276 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:20.250479 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:20.250586 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:20.251661 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0810 03:25:20.251764 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:25:20.256000 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12669
I20260810 03:25:20.257144 12739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:20.372784 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 12669
I20260810 03:25:20.405972 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12752
I20260810 03:25:20.407233 12818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:20.522274 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 12752
I20260810 03:25:20.546402 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 12833
I20260810 03:25:20.547752 12898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:20.659020 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 12833
2026-08-10T03:25:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4998 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:25:20 dist-test-slave-btw8 krb5kdc[13055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:25:20 dist-test-slave-btw8 krb5kdc[13055](info): set up 2 sockets
Aug 10 03:25:20 dist-test-slave-btw8 krb5kdc[13055](info): commencing operation
krb5kdc: starting...
W20260810 03:25:21.917685 20443 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.002s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/test-user.keytab.
Aug 10 03:25:22 dist-test-slave-btw8 krb5kdc[13055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:25:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.246.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
Entry for principal kudu/127.19.246.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab.
I20260810 03:25:22.303649 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41417
--webserver_interface=127.19.246.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41417,127.19.246.253:39325,127.19.246.252:39095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:25:22.675202 13071 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:25:22.675694 13071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:22.675827 13071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:22.675963 13071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:22.683574 13071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:22.683681 13071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:22.683740 13071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:22.683854 13071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:22.683974 13071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:22.694551 13071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41417,127.19.246.253:39325,127.19.246.252:39095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.254.keytab
--principal=kudu/127.19.246.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.254:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.246.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:22.696151 13071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:22.697552 13071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:22.710770 13076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:22.710862 13079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:22.710753 13077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:22.712129 13071 server_base.cc:1034] running on GCE node
I20260810 03:25:22.713060 13071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:22.714705 13071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:22.715961 13071 hybrid_clock.cc:648] HybridClock initialized: now 1786332322715886 us; error 56 us; skew 500 ppm
I20260810 03:25:22.716305 13071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:22 dist-test-slave-btw8 krb5kdc[13055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332322, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:22.719791 13071 init.cc:377] Logged in from keytab as kudu/127.19.246.254@KRBTEST.COM (short username kudu)
I20260810 03:25:22.721480 13071 webserver.cc:460] Webserver started at http://127.19.246.254:36953/ using document root <none> and password file <none>
I20260810 03:25:22.722249 13071 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:22.722390 13071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:22.722848 13071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:22.726133 13071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/data/instance:
uuid: "383fcf66fbc44e58a0fc034ecc958293"
format_stamp: "Formatted at 2026-08-10 03:25:22 on dist-test-slave-btw8"
server_key: "87b02bda27c4c25b524594c1e02a1f30"
server_key_iv: "761efa6e3ae4c5e752c4f8991347e966"
server_key_version: "encryptionkey@0"
I20260810 03:25:22.727013 13071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/wal/instance:
uuid: "383fcf66fbc44e58a0fc034ecc958293"
format_stamp: "Formatted at 2026-08-10 03:25:22 on dist-test-slave-btw8"
server_key: "87b02bda27c4c25b524594c1e02a1f30"
server_key_iv: "761efa6e3ae4c5e752c4f8991347e966"
server_key_version: "encryptionkey@0"
I20260810 03:25:22.733017 13071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:25:22.737177 13086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:22.738871 13071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:25:22.739117 13071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/wal
uuid: "383fcf66fbc44e58a0fc034ecc958293"
format_stamp: "Formatted at 2026-08-10 03:25:22 on dist-test-slave-btw8"
server_key: "87b02bda27c4c25b524594c1e02a1f30"
server_key_iv: "761efa6e3ae4c5e752c4f8991347e966"
server_key_version: "encryptionkey@0"
I20260810 03:25:22.739423 13071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:22.763622 13071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:22.768766 13071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:22.769104 13071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:22.791332 13071 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.254:41417
I20260810 03:25:22.791339 13144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.254:41417 every 8 connection(s)
I20260810 03:25:22.792937 13071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/data/info.pb
I20260810 03:25:22.797025 13145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:22.803490 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 13071
I20260810 03:25:22.803747 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-0/wal/instance
I20260810 03:25:22.804392 20443 external_mini_cluster.cc:1428] Setting key ad9a01f00deee871786fbeebca00351a
I20260810 03:25:22.804863 13145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:22 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332322, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:25:22.831175 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:22.809354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.19.246.254:41417)
0810 03:25:22.812448 (+ 3094us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:22.812462 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:22.812515 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:25:22.814181 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:22.814187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:22.814453 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:22.814665 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:22.816141 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:22.817385 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:22.818849 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:22.819513 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:22.819727 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:22.823276 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:22.823285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:22.823298 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:22.823332 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:22.826388 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:22.827226 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:22.827234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:22.827236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:22.827449 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:22.827848 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:22.827856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:22.827859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:22.828397 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:22.828612 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:22.829163 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 03:25:22.829619 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":2853,"spinlock_wait_cycles":17664,"thread_start_us":2627,"threads_started":1}
I20260810 03:25:22.831194 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:22.809354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41417 (local address 127.0.0.1:38108)
0810 03:25:22.811521 (+ 2167us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:22.811578 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:22.813762 (+ 2184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:22.814689 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:22.814696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:22.815554 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:22.815912 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:22.815930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:22.817565 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:22.817576 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:22.818695 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:22.818709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:22.819771 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:22.819778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:22.819959 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:22.820705 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:22.820728 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:22.823025 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:22.826629 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:22.826643 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:22.826658 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:22.827073 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:22.827558 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:22.827564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:22.827568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:22.827741 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:22.828554 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:22.828562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:22.828775 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 03:25:22.829037 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 03:25:22.829756 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1810,"mutex_wait_us":191,"thread_start_us":1558,"threads_started":1}
I20260810 03:25:22.834985 13145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:25:22.836313 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:22.835836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37418)
0810 03:25:22.836034 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:22.836167 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 03:25:22.836726 13089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
I20260810 03:25:22.838626 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:22.838183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37422)
0810 03:25:22.838378 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:22.838459 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 03:25:22.839026 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:39325: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.19.246.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab.
Entry for principal kudu/127.19.246.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab.
I20260810 03:25:22.888302 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.253:39325
--webserver_interface=127.19.246.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab
--principal=kudu/127.19.246.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41417,127.19.246.253:39325,127.19.246.252:39095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:25:22.888540 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } attempt: 1
I20260810 03:25:22.890010 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:22.889350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37428)
0810 03:25:22.889617 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:22.889806 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 03:25:22.891290 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:22.890774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37440)
0810 03:25:22.890964 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:22.891067 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 03:25:22.891597 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:39325: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
I20260810 03:25:22.959900 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } attempt: 2
I20260810 03:25:22.961278 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:22.960748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37442)
0810 03:25:22.961015 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:22.961110 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 03:25:22.962527 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:22.962122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37458)
0810 03:25:22.962306 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:22.962389 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 03:25:22.962747 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:39325: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
I20260810 03:25:23.054028 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } attempt: 3
I20260810 03:25:23.055212 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.054766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37464)
0810 03:25:23.054973 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.055076 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 03:25:23.056270 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.055907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37478)
0810 03:25:23.056054 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.056133 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 03:25:23.056442 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:39325: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
I20260810 03:25:23.199743 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } attempt: 4
I20260810 03:25:23.201401 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.200694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37492)
0810 03:25:23.201070 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.201189 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260810 03:25:23.202837 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.202375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37500)
0810 03:25:23.202600 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.202699 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 03:25:23.203186 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.253:39325: Network error: Client connection negotiation failed: client connection to 127.19.246.253:39325: connect: Connection refused (error 111)
W20260810 03:25:23.258786 13152 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:25:23.259207 13152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:23.259322 13152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:23.259473 13152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:23.267084 13152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:23.267194 13152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:23.267302 13152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:23.267407 13152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:23.267534 13152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:23.278216 13152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41417,127.19.246.253:39325,127.19.246.252:39095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.253.keytab
--principal=kudu/127.19.246.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.253:39325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.246.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:23.279685 13152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:23.281087 13152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:25:23.293900 13162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:23.293968 13159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:23.293994 13160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:23.295006 13152 server_base.cc:1034] running on GCE node
I20260810 03:25:23.295977 13152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:23.298274 13152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:23.299561 13152 hybrid_clock.cc:648] HybridClock initialized: now 1786332323299507 us; error 82 us; skew 500 ppm
I20260810 03:25:23.299979 13152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:23 dist-test-slave-btw8 krb5kdc[13055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:23.303686 13152 init.cc:377] Logged in from keytab as kudu/127.19.246.253@KRBTEST.COM (short username kudu)
I20260810 03:25:23.305429 13152 webserver.cc:460] Webserver started at http://127.19.246.253:43967/ using document root <none> and password file <none>
I20260810 03:25:23.306213 13152 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:23.306357 13152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:23.306722 13152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:23.310060 13152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/data/instance:
uuid: "5c0e853417ee473f87c92beabe3c49ea"
format_stamp: "Formatted at 2026-08-10 03:25:23 on dist-test-slave-btw8"
server_key: "b0df9453bd84931c9802dfb2415da2ba"
server_key_iv: "3e25aa0a7e9a2f7df786f82505f1d0af"
server_key_version: "encryptionkey@0"
I20260810 03:25:23.310993 13152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/wal/instance:
uuid: "5c0e853417ee473f87c92beabe3c49ea"
format_stamp: "Formatted at 2026-08-10 03:25:23 on dist-test-slave-btw8"
server_key: "b0df9453bd84931c9802dfb2415da2ba"
server_key_iv: "3e25aa0a7e9a2f7df786f82505f1d0af"
server_key_version: "encryptionkey@0"
I20260810 03:25:23.316817 13152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 03:25:23.320878 13170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:23.322582 13152 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:25:23.322830 13152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/wal
uuid: "5c0e853417ee473f87c92beabe3c49ea"
format_stamp: "Formatted at 2026-08-10 03:25:23 on dist-test-slave-btw8"
server_key: "b0df9453bd84931c9802dfb2415da2ba"
server_key_iv: "3e25aa0a7e9a2f7df786f82505f1d0af"
server_key_version: "encryptionkey@0"
I20260810 03:25:23.323094 13152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:23.380139 13152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:23.385306 13152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:23.385638 13152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:23.409052 13152 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.253:39325
I20260810 03:25:23.409081 13223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.253:39325 every 8 connection(s)
I20260810 03:25:23.410588 13152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/data/info.pb
I20260810 03:25:23.414686 13224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:23.419730 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 13152
I20260810 03:25:23.419955 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-1/wal/instance
I20260810 03:25:23.420532 20443 external_mini_cluster.cc:1428] Setting key 9af5be7997aeb936b228f5986b778890
I20260810 03:25:23.422765 13224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:23 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:25:23.450314 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.426715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.19.246.254:41417)
0810 03:25:23.427372 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:23.427383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:23.428570 (+ 1187us) server_negotiation.cc:407] Connection header received
0810 03:25:23.430753 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:23.430762 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:23.430868 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:23.431005 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:23.432842 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.434063 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.435864 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.436619 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.436757 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:23.441804 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:23.441814 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:23.441821 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:23.441885 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:23.446064 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:23.447043 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:23.447052 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:23.447059 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:23.447226 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:23.447909 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:23.447917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:23.447923 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:23.448408 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:23.448571 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:23.449498 (+ 927us) server_negotiation.cc:299] Negotiation successful
0810 03:25:23.449803 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":302,"threads_started":1}
I20260810 03:25:23.450918 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.427483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41417 (local address 127.0.0.1:38120)
0810 03:25:23.428379 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.428427 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:23.430446 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:23.431268 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:23.431277 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:23.431977 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:23.432595 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:23.432614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.434269 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.434274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:23.435671 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:23.435691 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.436956 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.436966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:23.437359 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:23.438276 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:23.438314 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:23.441432 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:23.446284 (+ 4852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:23.446292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:23.446309 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:23.446865 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:23.447558 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:23.447562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:23.447565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:23.447766 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:23.448645 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:23.448652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:23.449068 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 03:25:23.449327 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 03:25:23.449752 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":373,"threads_started":1}
I20260810 03:25:23.452852 13224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.246.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.246.252@KRBTEST.COM" created.
Aug 10 03:25:23 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:25:23.471323 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.453542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37514)
0810 03:25:23.453723 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.453737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:23.453824 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:23.455065 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:23.455073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:23.455651 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:23.455890 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:23.455905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.457349 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.457356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:23.458877 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:23.458894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.459922 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.459927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:23.460075 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:23.460779 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:23.460797 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:23.464151 (+ 3354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:23.468360 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:23.468366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:23.468370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:23.468792 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:23.469266 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:23.469272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:23.469276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:23.469444 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:23.470163 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:23.470171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:23.470353 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 03:25:23.470513 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 03:25:23.470854 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 03:25:23.471457 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.453872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.19.246.253:39325)
0810 03:25:23.454552 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:23.454558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:23.454580 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:25:23.454624 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:23.454630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:23.454851 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:23.455025 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:23.456097 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.457188 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.459045 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.459756 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.459856 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:23.464406 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:23.464422 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:23.464438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:23.464468 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:23.468152 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:23.468970 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:23.468978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:23.468980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:23.469135 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:23.469595 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:23.469599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:23.469601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:23.470006 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:23.470270 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:23.470683 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 03:25:23.470988 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":533,"thread_start_us":401,"threads_started":1}
I20260810 03:25:23.474304 13224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:25:23.475838 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.475196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35042)
0810 03:25:23.475474 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.475652 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260810 03:25:23.476181 13172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.477875 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.477508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35058)
0810 03:25:23.477657 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.477759 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 03:25:23.478123 13224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.502508 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } attempt: 5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.246.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab.
Entry for principal kudu/127.19.246.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab.
I20260810 03:25:23.508242 20443 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
/tmp/dist-test-taskZHDeAK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.252:39095
--webserver_interface=127.19.246.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab
--principal=kudu/127.19.246.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.246.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.246.254:41417,127.19.246.253:39325,127.19.246.252:39095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:25:23 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332322, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:25:23.518458 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.503435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37526)
0810 03:25:23.503681 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.503700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:23.503865 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:23.504247 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:23.504250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:23.504629 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:23.504857 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:23.504885 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.505934 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.505939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:23.506847 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:23.506863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.507873 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.507878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:23.508058 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:23.508845 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:23.508864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:23.510952 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:23.515712 (+ 4760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:23.515718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:23.515721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:23.516181 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:23.516645 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:23.516649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:23.516651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:23.516833 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:23.517530 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:23.517535 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:23.517708 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 03:25:23.517864 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 03:25:23.518173 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 03:25:23.518682 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.503732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.19.246.253:39325)
0810 03:25:23.503907 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:23.503911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:23.503930 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:25:23.504002 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:23.504006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:23.504087 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:23.504207 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:23.505033 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.505798 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.507012 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:23.507673 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:23.507826 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:23.511182 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:23.511192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:23.511208 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:23.511271 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:23.515464 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:23.516321 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:23.516329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:23.516335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:23.516495 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:23.516952 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:23.516960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:23.516965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:23.517372 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:23.517529 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:23.518003 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 03:25:23.518265 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:25:23.520004 13145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:25:23.521282 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.520759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35068)
0810 03:25:23.520974 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.521098 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 03:25:23.522403 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.522019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35080)
0810 03:25:23.522201 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.522278 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 03:25:23.522703 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.527513 13224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 1
I20260810 03:25:23.528890 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.528253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35086)
0810 03:25:23.528520 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.528668 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 03:25:23.530206 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.529664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35100)
0810 03:25:23.529893 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.530020 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 03:25:23.530546 13224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.573984 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 1
I20260810 03:25:23.575477 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.574793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35116)
0810 03:25:23.575053 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.575189 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 03:25:23.576869 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.576383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35122)
0810 03:25:23.576587 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.576699 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 03:25:23.577183 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.598821 13224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 2
I20260810 03:25:23.600265 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.599681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35130)
0810 03:25:23.599924 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.600053 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 03:25:23.601495 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.601103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35132)
0810 03:25:23.601268 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.601357 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 03:25:23.601756 13224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.645453 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 2
I20260810 03:25:23.646885 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.646220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35148)
0810 03:25:23.646510 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.646636 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 03:25:23.648363 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.647894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35150)
0810 03:25:23.648092 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.648206 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 03:25:23.648648 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.693040 13224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 3
I20260810 03:25:23.694666 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.694035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35166)
0810 03:25:23.694298 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.694423 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 03:25:23.696293 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.695601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35176)
0810 03:25:23.695933 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.696069 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260810 03:25:23.696588 13224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.754956 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 3
I20260810 03:25:23.756646 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.755940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35188)
0810 03:25:23.756236 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.756376 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 03:25:23.758013 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.757519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35196)
0810 03:25:23.757755 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.757861 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260810 03:25:23.758358 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.839855 13224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 4
I20260810 03:25:23.841207 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.840683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35200)
0810 03:25:23.840914 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.841012 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 03:25:23.842339 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.841959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35204)
0810 03:25:23.842149 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.842223 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 03:25:23.842634 13224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
W20260810 03:25:23.895819 13232 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:25:23.896194 13232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:25:23.896360 13232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:25:23.896483 13232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:25:23.905467 13232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:25:23.905642 13232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:25:23.905779 13232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:25:23.905872 13232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:25:23.905982 13232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:25:23.920830 13232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.246.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.246.254:41417,127.19.246.253:39325,127.19.246.252:39095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/krb5kdc/kudu_127.19.246.252.keytab
--principal=kudu/127.19.246.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.246.252:39095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.246.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 02:17:12 UTC on ccycloud.quasar-zbrcpm.root.comops.site
build id 4576981
ASAN enabled
I20260810 03:25:23.922290 13232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:25:23.923851 13232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:25:23.935621 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 4
I20260810 03:25:23.937131 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.936504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35212)
0810 03:25:23.936816 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.936926 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 03:25:23.937192 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:23.937389 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:25:23.937737 13242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:25:23.938331 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:23.937992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35222)
0810 03:25:23.938128 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:23.938209 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 03:25:23.938323 13232 server_base.cc:1034] running on GCE node
W20260810 03:25:23.938608 13145 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.246.252:39095: Network error: Client connection negotiation failed: client connection to 127.19.246.252:39095: connect: Connection refused (error 111)
I20260810 03:25:23.939325 13232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:25:23.941078 13232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:25:23.942373 13232 hybrid_clock.cc:648] HybridClock initialized: now 1786332323942269 us; error 60 us; skew 500 ppm
I20260810 03:25:23.942821 13232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 03:25:23 dist-test-slave-btw8 krb5kdc[13055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:25:23.946866 13232 init.cc:377] Logged in from keytab as kudu/127.19.246.252@KRBTEST.COM (short username kudu)
I20260810 03:25:23.948645 13232 webserver.cc:460] Webserver started at http://127.19.246.252:46763/ using document root <none> and password file <none>
I20260810 03:25:23.949404 13232 fs_manager.cc:359] Metadata directory not provided
I20260810 03:25:23.949501 13232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:25:23.949849 13232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:25:23.953133 13232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/data/instance:
uuid: "3148572d7a224a7fbd392deb22d5b360"
format_stamp: "Formatted at 2026-08-10 03:25:23 on dist-test-slave-btw8"
server_key: "7ce335db0fe8195aaa0ccb19da0ec238"
server_key_iv: "0c0531b76e4091d960dfb4c01df77c3a"
server_key_version: "encryptionkey@0"
I20260810 03:25:23.954000 13232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/wal/instance:
uuid: "3148572d7a224a7fbd392deb22d5b360"
format_stamp: "Formatted at 2026-08-10 03:25:23 on dist-test-slave-btw8"
server_key: "7ce335db0fe8195aaa0ccb19da0ec238"
server_key_iv: "0c0531b76e4091d960dfb4c01df77c3a"
server_key_version: "encryptionkey@0"
I20260810 03:25:23.959753 13232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 03:25:23.963507 13249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:25:23.965226 13232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:25:23.965477 13232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/data,/tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/wal
uuid: "3148572d7a224a7fbd392deb22d5b360"
format_stamp: "Formatted at 2026-08-10 03:25:23 on dist-test-slave-btw8"
server_key: "7ce335db0fe8195aaa0ccb19da0ec238"
server_key_iv: "0c0531b76e4091d960dfb4c01df77c3a"
server_key_version: "encryptionkey@0"
I20260810 03:25:23.965754 13232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:25:23.985553 13232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:25:23.990794 13232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:25:23.991137 13232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:25:24.015219 13232 rpc_server.cc:307] RPC server started. Bound to: 127.19.246.252:39095
I20260810 03:25:24.015239 13306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.246.252:39095 every 8 connection(s)
I20260810 03:25:24.017130 13232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/data/info.pb
I20260810 03:25:24.021188 20443 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu as pid 13232
I20260810 03:25:24.021243 13307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:25:24.021456 20443 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZHDeAK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786331905428759-20443-0/minicluster-data/master-2/wal/instance
I20260810 03:25:24.022101 20443 external_mini_cluster.cc:1428] Setting key 56c91ff125c233708026e133f024e812
I20260810 03:25:24.031562 13307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:24 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
Aug 10 03:25:24 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:25:24.049158 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.024502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.19.246.252:39095)
0810 03:25:24.025554 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.025566 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.025611 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:25:24.028181 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.028210 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.028678 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.029036 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.030930 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.032402 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.034213 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.035488 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.035682 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.040314 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.040322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.040334 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.040376 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.043932 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.044875 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.044882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.044890 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.045038 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.045778 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.045781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.045783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.046332 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.046668 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.047187 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.047729 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":581,"threads_started":1}
I20260810 03:25:24.051227 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.034668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.19.246.254:41417)
0810 03:25:24.035344 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.035375 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.036000 (+ 625us) server_negotiation.cc:407] Connection header received
0810 03:25:24.036197 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.036201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.036288 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.036414 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.037552 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.038243 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.040016 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.040440 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.040587 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.044388 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.044397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.044401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.044462 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.048111 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.048918 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.048924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.048927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.049108 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.049715 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.049720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.049723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.050077 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.050258 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.050745 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.050931 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":407,"threads_started":1}
I20260810 03:25:24.051537 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.034799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.254:41417 (local address 127.0.0.1:38130)
0810 03:25:24.035823 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:24.035883 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:24.036057 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:24.036492 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:24.036499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:24.037141 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:24.037390 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.037398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.038397 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.038402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.039852 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.039864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.040660 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.040668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.040856 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.041579 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:24.041609 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:24.044170 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:24.048283 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.048293 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.048298 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.048762 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.049260 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.049264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.049266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.049481 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.050247 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:24.050253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:24.050418 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 03:25:24.050585 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 03:25:24.051044 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":568,"threads_started":1}
I20260810 03:25:24.053607 13307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:24 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
Aug 10 03:25:24 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.254@KRBTEST.COM
I20260810 03:25:24.068369 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.054380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.253:39325 (local address 127.0.0.1:37540)
0810 03:25:24.055463 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:24.055478 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:24.055564 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:24.055887 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:24.055890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:24.056223 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:24.056368 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.056375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.057428 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.057435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.058307 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.058318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.059227 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.059233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.059484 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.060083 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:24.060100 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:24.062338 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:24.065704 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.065710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.065714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.066119 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.066513 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.066518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.066522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.066682 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.067238 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:24.067242 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:24.067461 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 03:25:24.067673 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 03:25:24.067984 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922}
I20260810 03:25:24.068454 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.054525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.19.246.253:39325)
0810 03:25:24.055164 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.055169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.055549 (+ 380us) server_negotiation.cc:407] Connection header received
0810 03:25:24.055652 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.055657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.055738 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.055857 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.056561 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.057282 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.058453 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.059054 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.059211 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.062534 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.062540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.062543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.062577 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.065544 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.066247 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.066252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.066256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.066391 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.066796 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.066801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.066804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.067094 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.067260 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.067767 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.067988 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":334,"threads_started":1}
I20260810 03:25:24.069479 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.053080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.19.246.254:41417)
0810 03:25:24.053258 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.053262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.053365 (+ 103us) server_negotiation.cc:407] Connection header received
0810 03:25:24.053547 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.053551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.053720 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.053830 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.055342 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.056325 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.057616 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.058067 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.058203 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.063312 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.063318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.063322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.063415 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.066371 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.067196 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.067205 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.067209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.067461 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.067905 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.067911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.067914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.068232 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.068565 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.068981 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.069212 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:25:24.070050 13307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:25:24 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.252@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
Aug 10 03:25:24 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.246.253@KRBTEST.COM
I20260810 03:25:24.088161 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.071153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35248)
0810 03:25:24.071488 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:24.071504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:24.071651 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:24.072214 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:24.072219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:24.072695 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:24.072834 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.072843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.074497 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.074507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.076121 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.076135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.077290 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.077296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.077532 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.078302 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:24.078330 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:24.080874 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:24.085254 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.085263 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.085269 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.085752 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.086298 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.086306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.086310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.086513 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.087285 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:24.087288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:24.087501 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 03:25:24.087626 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 03:25:24.087873 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260810 03:25:24.088609 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.071136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.19.246.252:39095)
0810 03:25:24.071390 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.071400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.071614 (+ 214us) server_negotiation.cc:407] Connection header received
0810 03:25:24.071858 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.071864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.071942 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.072010 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.073094 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.074333 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.076292 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.077050 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.077217 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.081234 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.081243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.081250 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.081305 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.085092 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.085919 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.085925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.085930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.086104 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.086698 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.086706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.086712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.087140 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.087328 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.087856 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.088125 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:25:24.092000 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.071728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.19.246.253:39325)
0810 03:25:24.071897 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.071903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.071956 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:25:24.072114 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.072119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.072237 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.072340 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.074429 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.075758 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.077582 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.078225 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.078366 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.084519 (+ 6153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.084525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.084528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.084578 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.088383 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.089504 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.089513 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.089517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.089749 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.090316 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.090320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.090322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.090700 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.090944 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.091460 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.091681 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:25:24.094671 13307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: Bootstrap starting.
I20260810 03:25:24.096993 20443 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:25:24.098249 13307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:24.099593 13307 log.cc:824] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:24.102507 13307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: No bootstrap required, opened a new log
I20260810 03:25:24.099536 13118 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "383fcf66fbc44e58a0fc034ecc958293"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38142
I20260810 03:25:24.112541 13307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } }
I20260810 03:25:24.113036 13307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3148572d7a224a7fbd392deb22d5b360, State: Initialized, Role: FOLLOWER
I20260810 03:25:24.113571 13307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } }
I20260810 03:25:24.114583 13317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.114941 13317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:24.116027 13307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:25:24.127094 13328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:25:24.127261 13328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:25:24.124665 13232 master_runner.cc:186] Error getting master registration for 127.19.246.254:41417: OK, instance_id { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" instance_seqno: 1786332322785293 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:25:24.134306 13232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:25:24.141937 13224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 5
Aug 10 03:25:24 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332323, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.253@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
I20260810 03:25:24.161675 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.142884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.19.246.252:39095)
0810 03:25:24.143500 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.143507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.143532 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:25:24.143604 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.143612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.143702 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.143803 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.145036 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.146259 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.148326 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.149222 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.149387 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.154282 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.154288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.154291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.154330 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.158351 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.159212 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.159219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.159223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.159434 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.159923 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.159930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.159934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.160330 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.160518 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.160974 (+ 456us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.161224 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260810 03:25:24.161662 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.142783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35260)
0810 03:25:24.143004 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:24.143024 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:24.143194 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:24.143914 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:24.143921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:24.144549 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:24.144805 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.144821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.146452 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.146458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.148128 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.148148 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.149434 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.149441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.149734 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.150559 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:24.150588 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:24.154012 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:24.158554 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.158561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.158565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.159046 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.159583 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.159589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.159593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.159796 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.160483 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:24.160489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:24.160661 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 03:25:24.160824 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 03:25:24.161209 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 03:25:24.166834 13224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: Bootstrap starting.
I20260810 03:25:24.170647 13224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:24.172305 13224 log.cc:824] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:24.175760 13224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: No bootstrap required, opened a new log
I20260810 03:25:24.183765 13224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } }
I20260810 03:25:24.184149 13224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c0e853417ee473f87c92beabe3c49ea, State: Initialized, Role: FOLLOWER
I20260810 03:25:24.184700 13224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } }
I20260810 03:25:24.185830 13332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.186244 13332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:24.187126 13224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:25:24.198376 13343 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:25:24.198583 13343 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:25:24.195170 13152 master_runner.cc:186] Error getting master registration for 127.19.246.254:41417: OK, instance_id { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" instance_seqno: 1786332322785293 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:25:24.208220 13152 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:25:24.215848 13145 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } attempt: 5
Aug 10 03:25:24 dist-test-slave-btw8 krb5kdc[13055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786332322, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.246.254@KRBTEST.COM for kudu/127.19.246.252@KRBTEST.COM
I20260810 03:25:24.236994 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.216971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.19.246.252:39095)
0810 03:25:24.217645 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.217657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.217677 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:25:24.217743 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.217749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.217878 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.217976 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.219300 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.220514 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.222457 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.223323 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.223494 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.228566 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.228574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.228580 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.228639 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.232877 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.233999 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.234007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.234011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.234323 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.234891 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.234897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.234900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.235508 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.235714 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.236297 (+ 583us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.236581 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530}
I20260810 03:25:24.237028 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.216909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.246.252:39095 (local address 127.0.0.1:35264)
0810 03:25:24.217230 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 03:25:24.217253 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:25:24.217395 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:25:24.218077 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:25:24.218084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:25:24.218741 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:25:24.219124 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.219141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.220683 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.220690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.222233 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:25:24.222257 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.223540 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.223549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:25:24.223778 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.224785 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:25:24.224817 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:25:24.228189 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:25:24.233134 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.233141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.233146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.233819 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.234488 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:25:24.234494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:25:24.234498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:25:24.234731 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:25:24.235702 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:25:24.235709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:25:24.235973 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 03:25:24.236177 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 03:25:24.236527 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
W20260810 03:25:24.243826 13255 tablet.cc:2369] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:24.244832 13145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293: Bootstrap starting.
I20260810 03:25:24.249341 13145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293: Neither blocks nor log segments found. Creating new log.
I20260810 03:25:24.250829 13145 log.cc:824] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293: Log is configured to *not* fsync() on all Append() calls
I20260810 03:25:24.254936 13145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293: No bootstrap required, opened a new log
I20260810 03:25:24.263489 13145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } }
I20260810 03:25:24.263973 13145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383fcf66fbc44e58a0fc034ecc958293, State: Initialized, Role: FOLLOWER
I20260810 03:25:24.264546 13145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } }
I20260810 03:25:24.266098 13346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.266973 13346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:24.267233 13145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:25:24.267338 13118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 03:25:24.267707 13118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:24.271884 13118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } }
W20260810 03:25:24.272825 13095 tablet.cc:2369] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:24.276294 13198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "383fcf66fbc44e58a0fc034ecc958293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5c0e853417ee473f87c92beabe3c49ea"
I20260810 03:25:24.276875 13198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:24.281075 13198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383fcf66fbc44e58a0fc034ecc958293 in term 1.
I20260810 03:25:24.282050 13089 leader_election.cc:302] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383fcf66fbc44e58a0fc034ecc958293, 5c0e853417ee473f87c92beabe3c49ea; no voters:
I20260810 03:25:24.286393 13279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "383fcf66fbc44e58a0fc034ecc958293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3148572d7a224a7fbd392deb22d5b360"
I20260810 03:25:24.287024 13279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:25:24.292299 13279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383fcf66fbc44e58a0fc034ecc958293 in term 1.
I20260810 03:25:24.293756 13118 leader_election.cc:288] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [CANDIDATE]: Term 1 election: Requested vote from peers 5c0e853417ee473f87c92beabe3c49ea (127.19.246.253:39325), 3148572d7a224a7fbd392deb22d5b360 (127.19.246.252:39095)
I20260810 03:25:24.297027 13346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:25:24.297899 13346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [term 1 LEADER]: Becoming Leader. State: Replica: 383fcf66fbc44e58a0fc034ecc958293, State: Running, Role: LEADER
I20260810 03:25:24.299062 13346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } }
I20260810 03:25:24.333379 13349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 383fcf66fbc44e58a0fc034ecc958293. Latest consensus state: current_term: 1 leader_uuid: "383fcf66fbc44e58a0fc034ecc958293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.334017 13349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:24.340081 13359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:25:24.358963 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.327756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.19.246.254:41417)
0810 03:25:24.327935 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.327941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.337290 (+ 9349us) server_negotiation.cc:407] Connection header received
0810 03:25:24.338776 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.338808 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.338931 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.339047 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.341895 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.343117 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.345629 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.346308 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.346408 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.350124 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.350131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.350136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.350184 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.354539 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.355829 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.355837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.355841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.356056 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.356612 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.356618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.356622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.357067 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.357446 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.358116 (+ 670us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.358446 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:25:24.373355 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.326597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.19.246.252:39095)
0810 03:25:24.326855 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.326862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.332592 (+ 5730us) server_negotiation.cc:407] Connection header received
0810 03:25:24.332848 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.332854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.332973 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.333097 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.335911 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.337051 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.344782 (+ 7731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.345564 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.345687 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.363553 (+ 17866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.363562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.363568 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.363623 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.368194 (+ 4571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.369895 (+ 1701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.369902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.369906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.370082 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.370906 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.370912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.370916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.371400 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.371662 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.372629 (+ 967us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.372929 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:25:24.377424 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:24.337897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.19.246.253:39325)
0810 03:25:24.338745 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.338751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.340947 (+ 2196us) server_negotiation.cc:407] Connection header received
0810 03:25:24.346998 (+ 6051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.347005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.347119 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.347202 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.359512 (+ 12310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.360912 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.363170 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:24.363983 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:24.364146 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:24.368633 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:25:24.368643 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:25:24.368649 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:25:24.368702 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:25:24.372985 (+ 4283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.374307 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.374314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.374318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.374631 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:25:24.375153 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:25:24.375161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:25:24.375165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:25:24.375644 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:25:24.375980 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:24.376677 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 03:25:24.376969 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686}
I20260810 03:25:24.359320 13359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 03:25:24.388684 13176 tablet.cc:2369] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:25:24.395944 13279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [term 1 FOLLOWER]: Refusing update from remote peer 383fcf66fbc44e58a0fc034ecc958293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:24.398386 13198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [term 1 FOLLOWER]: Refusing update from remote peer 383fcf66fbc44e58a0fc034ecc958293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:25:24.399802 13349 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:24.401044 13349 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:25:24.440562 13317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 383fcf66fbc44e58a0fc034ecc958293. Latest consensus state: current_term: 1 leader_uuid: "383fcf66fbc44e58a0fc034ecc958293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.440922 13317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:24.446928 13368 mvcc.cc:205] Tried to move back new op lower bound from 7316817200709242880 to 7316817200349143040. Current Snapshot: MvccSnapshot[applied={T|T < 7316817200709242880}]
I20260810 03:25:24.453589 13372 mvcc.cc:205] Tried to move back new op lower bound from 7316817200709242880 to 7316817200349143040. Current Snapshot: MvccSnapshot[applied={T|T < 7316817200709242880}]
I20260810 03:25:24.458398 13346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "383fcf66fbc44e58a0fc034ecc958293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.458770 13346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:24.460449 13359 catalog_manager.cc:1347] Generated new cluster ID: f762e36be6c34a1e8c680d17a40423cd
I20260810 03:25:24.460590 13359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:25:24.464730 13317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "383fcf66fbc44e58a0fc034ecc958293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.464983 13317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:24.465296 13332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 383fcf66fbc44e58a0fc034ecc958293. Latest consensus state: current_term: 1 leader_uuid: "383fcf66fbc44e58a0fc034ecc958293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.465644 13332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:24.468982 13349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "383fcf66fbc44e58a0fc034ecc958293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.469293 13349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293 [sys.catalog]: This master's current role is: LEADER
I20260810 03:25:24.471395 13373 mvcc.cc:205] Tried to move back new op lower bound from 7316817200709242880 to 7316817200349143040. Current Snapshot: MvccSnapshot[applied={T|T < 7316817200709242880}]
I20260810 03:25:24.485093 13332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "383fcf66fbc44e58a0fc034ecc958293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383fcf66fbc44e58a0fc034ecc958293" member_type: VOTER last_known_addr { host: "127.19.246.254" port: 41417 } } peers { permanent_uuid: "5c0e853417ee473f87c92beabe3c49ea" member_type: VOTER last_known_addr { host: "127.19.246.253" port: 39325 } } peers { permanent_uuid: "3148572d7a224a7fbd392deb22d5b360" member_type: VOTER last_known_addr { host: "127.19.246.252" port: 39095 } } }
I20260810 03:25:24.485400 13332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:25:24.521634 13359 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:25:24.523785 13359 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:25:24.547216 13359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 383fcf66fbc44e58a0fc034ecc958293: Generated new TSK 0
I20260810 03:25:24.548475 13359 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:25:24.707252 13311 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.692635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.19.246.254:41417)
0810 03:25:24.692910 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.692917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.704334 (+ 11417us) server_negotiation.cc:407] Connection header received
0810 03:25:24.704847 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.704858 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.704996 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.705110 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.706938 (+ 1828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38158: BlockingRecv error: recv got EOF from 127.0.0.1:38158 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 03:25:24.710346 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.707970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.19.246.253:39325)
0810 03:25:24.708185 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.708191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.708481 (+ 290us) server_negotiation.cc:407] Connection header received
0810 03:25:24.708737 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.708743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.708825 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.708909 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.710060 (+ 1151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37566: BlockingRecv error: recv got EOF from 127.0.0.1:37566 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 03:25:24.710361 13310 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.694548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.19.246.252:39095)
0810 03:25:24.694803 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.694814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.707323 (+ 12509us) server_negotiation.cc:407] Connection header received
0810 03:25:24.708673 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.708684 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.708782 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.708865 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.710075 (+ 1210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35278: BlockingRecv error: recv got EOF from 127.0.0.1:35278 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 03:25:24.759760 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.748671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.19.246.253:39325)
0810 03:25:24.748933 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.748942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.756826 (+ 7884us) server_negotiation.cc:407] Connection header received
0810 03:25:24.757553 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.757561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.757740 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.757855 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.759411 (+ 1556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37568: BlockingRecv error: recv got EOF from 127.0.0.1:37568 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260810 03:25:24.804332 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.794163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.19.246.253:39325)
0810 03:25:24.794378 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.794389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.801919 (+ 7530us) server_negotiation.cc:407] Connection header received
0810 03:25:24.802477 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.802485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.802596 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.802710 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.803982 (+ 1272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37578: BlockingRecv error: recv got EOF from 127.0.0.1:37578 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 03:25:24.856773 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.842271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.19.246.253:39325)
0810 03:25:24.842539 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.842550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.854188 (+ 11638us) server_negotiation.cc:407] Connection header received
0810 03:25:24.854892 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.854900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.855019 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.855115 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.856411 (+ 1296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37580: BlockingRecv error: recv got EOF from 127.0.0.1:37580 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260810 03:25:24.900900 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.898220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.19.246.253:39325)
0810 03:25:24.898481 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.898489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.899205 (+ 716us) server_negotiation.cc:407] Connection header received
0810 03:25:24.899442 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.899449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.899558 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.899707 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.900631 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37596: BlockingRecv error: recv got EOF from 127.0.0.1:37596 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 03:25:24.920117 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.917561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.19.246.253:39325)
0810 03:25:24.917823 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.917832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.918250 (+ 418us) server_negotiation.cc:407] Connection header received
0810 03:25:24.918544 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.918549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.918651 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.918785 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.919803 (+ 1018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44906: BlockingRecv error: recv got EOF from 127.0.0.1:44906 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260810 03:25:24.942308 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.940253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.19.246.253:39325)
0810 03:25:24.940453 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.940460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.940812 (+ 352us) server_negotiation.cc:407] Connection header received
0810 03:25:24.940980 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.940986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.941081 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.941214 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.942069 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44912: BlockingRecv error: recv got EOF from 127.0.0.1:44912 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 03:25:24.980501 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:24.978438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.19.246.253:39325)
0810 03:25:24.978652 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:24.978659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:24.979012 (+ 353us) server_negotiation.cc:407] Connection header received
0810 03:25:24.979206 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:24.979211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:24.979317 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:24.979487 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:24.980294 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44914: BlockingRecv error: recv got EOF from 127.0.0.1:44914 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 03:25:25.050881 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:25.048591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.19.246.253:39325)
0810 03:25:25.048828 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:25.048832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:25.049242 (+ 410us) server_negotiation.cc:407] Connection header received
0810 03:25:25.049410 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:25.049414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:25.049505 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:25.049616 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:25.050558 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44930: BlockingRecv error: recv got EOF from 127.0.0.1:44930 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 03:25:25.129547 13328 catalog_manager.cc:1259] Loaded cluster ID: f762e36be6c34a1e8c680d17a40423cd
I20260810 03:25:25.129668 13328 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: loading cluster ID for follower catalog manager: success
I20260810 03:25:25.132261 13328 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: acquiring CA information for follower catalog manager: success
I20260810 03:25:25.134236 13328 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3148572d7a224a7fbd392deb22d5b360: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 03:25:25.184782 13315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:25:25.182578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.19.246.253:39325)
0810 03:25:25.182860 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:25.182869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:25.183214 (+ 345us) server_negotiation.cc:407] Connection header received
0810 03:25:25.183511 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:25.183516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:25.183598 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:25.183718 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:25:25.184506 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44942: BlockingRecv error: recv got EOF from 127.0.0.1:44942 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 03:25:25.201009 13343 catalog_manager.cc:1259] Loaded cluster ID: f762e36be6c34a1e8c680d17a40423cd
I20260810 03:25:25.201135 13343 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: loading cluster ID for follower catalog manager: success
I20260810 03:25:25.204005 13343 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: acquiring CA information for follower catalog manager: success
I20260810 03:25:25.206162 13343 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c0e853417ee473f87c92beabe3c49ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:25:25.451325 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:25.445301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.19.246.253:39325)
0810 03:25:25.445567 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:25.445573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:25.445821 (+ 248us) server_negotiation.cc:407] Connection header received
0810 03:25:25.446058 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:25.446062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:25.446252 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:25.446391 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:25.446887 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.447652 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.449057 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.449487 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.449638 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:25.450159 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:25.450697 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:25.450863 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:25.450924 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 03:25:25.451013 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 03:25:25.465124 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:25.458474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.19.246.252:39095)
0810 03:25:25.459026 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:25.459036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:25.459060 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:25:25.459403 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:25.459410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:25.459606 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:25.459811 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:25.460266 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.461098 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.462580 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.463028 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.463201 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:25.463814 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:25.464390 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:25.464570 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:25.464677 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 03:25:25.464773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":264,"threads_started":1}
I20260810 03:25:25.487290 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:25.472712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.19.246.252:39095)
0810 03:25:25.472951 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:25.472962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:25.479823 (+ 6861us) server_negotiation.cc:407] Connection header received
0810 03:25:25.480201 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:25.480211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:25.480344 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:25.480428 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:25.481438 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.482495 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.484915 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.485530 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.485667 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:25.486311 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:25.486464 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:25.486539 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:25.486803 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 03:25:25.486899 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:25:25.487751 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:25.472618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.19.246.254:41417)
0810 03:25:25.479190 (+ 6572us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:25.479195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:25.479217 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:25:25.479332 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:25.479336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:25.479591 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:25.479833 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:25.480819 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.482140 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.484635 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.485414 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.485581 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:25.486249 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:25.486710 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:25.486824 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:25.487259 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 03:25:25.487406 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6452,"thread_start_us":6348,"threads_started":1}
I20260810 03:25:25.488023 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:25:25.480005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.19.246.253:39325)
0810 03:25:25.480253 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:25:25.480259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:25:25.481017 (+ 758us) server_negotiation.cc:407] Connection header received
0810 03:25:25.481262 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:25:25.481267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:25:25.481386 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:25:25.481491 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:25:25.482047 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.483050 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.485337 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:25:25.486052 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:25:25.486200 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:25:25.486777 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:25:25.486935 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:25:25.487055 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:25:25.487302 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 03:25:25.487679 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 03:25:25.493913 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 13071
I20260810 03:25:25.495612 13140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:25.609504 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 13071
I20260810 03:25:25.637034 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 13152
I20260810 03:25:25.638475 13219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:25.752724 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 13152
I20260810 03:25:25.777746 20443 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu pid 13232
I20260810 03:25:25.778982 13301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 03:25:25.890465 20443 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZHDeAK/build/asan/bin/kudu with pid 13232
2026-08-10T03:25:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5236 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10235 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (420392 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST